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Sheila Fraser, FCA 
Auditor General of Canada 


Management of Forensic 
Laboratory Services 


A Message from 
the Auditor General of Canada— 
May 2007 


In the reports we present to Parliament, we provide answers to some 
important questions about the management of government programs 
and services. Are they managed with due regard to economy, 
efficiency, and their impact on the environment? And does the 
government have measures in place to determine if programs and 
services are effective in achieving the objectives set for them? 


Our audit mandate covers the full range of activities of the federal 
government. Given the size of government and the scope of its 
activities, deciding what to audit is a challenging exercise. We may 
decide to audit an activity carried out in a single department or in 
several; we may select an issue that affects many departments. 
Choosing audit subjects well is essential to producing reports that are 
useful to Parliament. 


We select areas for audit based partly on the risks they may involve. 
Examples of high-risk areas include those that cost taxpayers 
significant amounts of money or that involve public safety or national 
security. We may decide to audit an area that offers potential for 
improving government results or is of great interest to Parliament and 
the Canadian public. We pay particular attention to requests by 
parliamentary committees for audits of specific areas of concern to 
them, and we try to incorporate such requests into our audit planning 
where our resources and mandate permit. 


This report includes two chapters on audits performed at the request of 
standing committees of the House of Commons. We audited the 
management of the RCMP’s Forensic Laboratory Services, following a 
request by the Standing Committee on Justice, Human Rights, Public 
Safety and Emergency Preparedness. And we audited the Canadian 
Agricultural Income Stabilization program after receiving a request 
from the Standing Committee on Agriculture and Agri-Food. 


The RCMP’s Forensic Laboratory Services is the main provider of 
forensic analysis for most police agencies and courts in Canada. A 
national laboratory service operating at six sites in Halifax, Ottawa, 
Winnipeg, Regina, Edmonton, and Vancouver, it provides a range of 
forensic analysis services relating to criminal cases. Among its 
responsibilities is the operation of the national DNA data bank 
containing DNA samples from thousands of convicted offenders. 
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Canadian Agricultural Income 
Stabilization 


Forensic labs are an important element of the criminal justice system, 
since investigators and prosecutors rely on forensics to help identify or 
eliminate suspects and to provide evidence that can hold up in court. 
Delays in obtaining results of forensic analysis can slow police 
investigations and leave criminals on the street to reoffend. 


The request that we conduct this audit came after the Justice 
Committee heard conflicting testimony on the performance of the lab 
services and the status of DNA cases. At hearings of the Committee 
in 2004 and 2005, the RCMP had testified that its forensic lab services 
matched the best in the world, that the labs had no backlog of cases, 
that all cases involving violent crimes were given priority, and that the 
clients—police forces and prosecutors—were satisfied. Our audit 
found many of these claims to be incorrect. 


We found that while the lab services can process urgent requests in less 
than 15 days, only 1 percent of all requests are classed as urgent. 
Among the remaining 99 percent, categorized as routine requests, are 
cases of violent crimes such as murder, sexual assault, robbery, and 
abduction. For most routine requests, the lab service is unable to meet 
its own turnaround target of 30 days, partly due to its backlog of DNA 
cases. For instance, on average a request for DNA analysis took 

114 days to complete during the 2005-06 fiscal year. 


We also found problems in the quality control of lab results. In 
practice, there are significant weaknesses in how quality issues are 
defined, recorded, monitored, and resolved. We did not audit the 
quality of the forensic science used and therefore offer no opinion on 
the validity of the analysis or the results. However, we found that the 
lab services’ own quality management system had failed to identify 
problems with the new automated process for DNA analysis. After 
scientific staff questioned the validity and consistency of some DNA 
results, the automated process was shut down for several months. The 
difficulties continued, however, and lab officials did not identify the 
problem as a quality issue for more than a year. 


We last audited the Forensic Laboratory Services in 2000. At that 
time, management committed to address our recommendations within 
three years. Now, seven years later, we find it necessary to repeat most 
of those recommendations. 


We audited the Canadian Agricultural Income Stabilization (CAIS) 
program following a request by the House of Commons Agriculture 
and Agri-Food Committee. The CAIS program was the latest in a 
number of programs developed by Agriculture and Agri-Food Canada 


Report of the Auditor General of Canada—May 2007 


Human Resources Management— 
Foreign Affairs and International 
Trade Canada 


A MESSAGE FROM THE AUDITOR GENERAL OF CANADA 


to provide income support to agricultural producers when their farm 
income dropped due to circumstances beyond their control. Producers 
received a benefit payment based on information included with their 
applications to CAIS. 


We found that the Department was not transparent in explaining how 
it calculated benefits. When it changed some of the information 
submitted by producers, it did not explain in the final benefits 
statement the reasons for changes or their impact on the benefit 
amount. As a result, producers did not receive in an easily 
understandable form all the information they needed to decide 
whether the payment they received was appropriate or whether they 
should appeal an unfavourable decision. Further, there were long 
delays before producers learned whether they would receive a benefit 
and in what amount. 


We found that the Department focused on detecting overpayments to 
producers rather than possible underpayments. Given that the onus 
was on producers to identify and question incorrect figures in the 
statement sent to them by the Department, this approach was not 
consistent with the objective set out for the program in the Farm 
Income Protection Act—to protect the income of producers. 
Furthermore, we were surprised to discover that some of the staff 
involved in assessing applications were also acting as paid consultants 
to help producers prepare their applications. This practice contravenes 
the conflict-of-interest provisions in the Values and Ethics Code for the 
Public Service. We note that the Department has now instructed its 
employees not to do this. 


Since we completed our audit work, the government has announced 
its intention to change the CAIS program. We believe the Department 
should look for ways to simplify the delivery of farm income support 
and make the process more user-friendly. 


Canada relies on the people who work in its missions to be its “face 
abroad.” Foreign Affairs and International Trade Canada operates 

170 missions in 111 countries around the world. In light of increasing 
globalization, its employees abroad and at headquarters in Ottawa 
must respond to growing demands for services from other government 
departments, individual Canadians, and Canadian companies. The 
government also relies on the Department for timely advice to help it 
develop its foreign policy. 
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Federal Loans and Grants for 
Post-Secondary Education 


It is important to Canadians that the Department have the right 
people in the right place at the right time and that it be able to act 
quickly and decisively in a rapidly changing environment. 


We found that the Department has no strategic human resources plan. 
It does not have a complete picture of the people, competencies, and 
experience it needs now and in the future. In the next few years, more 
than half of its employees in the management category will be eligible 
to retire. The Department has been unable through its recruitment, 
promotion, and assignment processes to fill its positions in a timely way 
with enough people who have the required skills and competencies. 


We also found that the Department does not pay enough attention to 
the management of locally engaged staff in Canadian missions abroad. 
Although these employees account for half of its workforce, the 
Department does not have accurate and complete information about 
them. It is not giving its missions the support they need to properly 
manage the local staff they rely on. 


The Department has difficulty finding employees willing to accept 
postings to its missions in the United States or in dangerous parts of 
the world. The number of missions considered dangerous or extremely 
difficult has doubled since 1991. The process in place to compensate 
employees for living conditions abroad and provide incentives for 
hardship postings has not allowed the Department to respond in a 
timely way to changing circumstances. It has been working with the 
Treasury Board Secretariat to find solutions to these problems, but 
progress is slow. 


The Canada Student Loans Program of Human Resources and Social 
Development Canada (HRSDC) provides loans and grants to qualified 
students who need financial assistance for post-secondary education. 
In 2005-06, the Department loaned $1.9 billion to some 350,000 
post-secondary students and awarded 78,500 grants amounting to 
about $129 million. 


The grants provided by the Department and the Millennium Bursary 
and Access Bursary programs of the Canada Millennium Scholarship 
Foundation are designed to help limit the amount of debt that 
recipients accumulate. In 2006, the Foundation awarded almost 
$340 million in bursaries to over 100,000 students. 


Although there are gaps in measuring and reporting program results, 
we found that the Canada Student Loans Program is well managed. 
The Foundation also manages its bursary programs well, and it 


Report of the Auditor General of Canada—May 2007 


Modernizing the NORAD System in 
Canada 


A MESSAGE FROM THE AUDITOR GENERAL OF CANADA 


measures and reports the results adequately. Both organizations have 
appropriate controls for disbursing payments to students. 


The Canada Revenue Agency (CRA) contracts with private collection 
agencies to collect a large proportion of defaulted student loans on 
behalf of HRSDC. Both the CRA and HRSDC have a responsibility to 
monitor the private agencies’ compliance with specified collection 
requirements. However, we found that neither one carries out this 
monitoring adequately. 


HRSDC and the Foundation are taking steps to make prospective 
post-secondary students and their families more aware of the financial 
assistance available to them. The Department has also improved its 
communication to students about measures that would help them 
manage their debt. 


This is the second foundation we have examined since Parliament 
amended the Auditor General Act in 2005, extending our mandate to 
include them in performance audits of related federal programs. 


Canada shares the defence of North American airspace with the 
United States through the North American Aerospace Defense 
(NORAD) agreement. Together they operate an air surveillance and 
control system over North American airspace. In 1997, the 

two countries agreed to update and modernize this system. 


Originally, the Canadian government approved funding of about 

$93 million to define requirements and modernize the Canadian 
portion of the NORAD system. The project included a new system and 
the construction of an above-ground complex to house it, which would 
allow National Defence to close its underground complex at 

North Bay, Ontario. 


We found that the modernization project has taken longer and cost 
much more than planned. We estimate that the cost of replacing the 
NORAD system in Canada and providing a new above-ground 
complex will total about $156 million, almost double the original 
estimate. The cost of the modernization exceeded estimates because of 
the way the project was structured and managed. There were problems 
in project management and oversight and in the construction of the 
complex. The Department did not always inform the government in a 
timely way of changes in the project’s scope. 


The Department did not complete a review of building security 
requirements before construction of the above-ground complex and, at 
the time of our audit, questions remained about the security of the 
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Managing the Delivery of Legal 
Services to Government 


building and the feasibility of using it for the intended purpose. As a 
result, the Department has had to keep some operations in the 
underground complex. 


Though it originally expected to save $16 million a year in personnel 
and operating costs after closing the underground complex and 
operating out of the facility above ground, National Defence was still 
operating both facilities at the time of our audit. Anticipated savings 
have not yet been realized. 


Justice Canada can be characterized as Canada’s largest law firm, with 
about 2,500 lawyers and a budget of close to $1 billion in 2006-07. 
The Department provides legal advice to federal departments, drafts 
regulations and legislation, and with a few exceptions, represents all 
departments and agencies in court. Since our last audit in 1993, the 
complexity and volume of litigation have increased significantly, and 
the cost of the Department’s legal services has more than tripled. 


We found that the Department relies on a number of quality 
management practices and the expertise of the lawyers it employs to 
help ensure the quality of the legal services it provides to the 
government. However, it does not know whether it is delivering the 
right level of quality, because it lacks a departmental quality 
management system and has not articulated what it means by quality. 
This impedes its ability to deliver consistent quality of service to the 
government and to assess whether it is meeting that objective. 


The current cost-sharing arrangements with client departments 
provide little incentive to control costs and manage the demand for 
legal services—a problem Justice Canada has been aware of for a long 
time. We also found that it is inefficient to administer the current 
arrangements—there are more than 100 arrangements with the eight 
departments in our audit sample. Further, cost information provided to 
client departments is weak and inconsistent. Justice Canada also does 
not know whether it is delivering legal services cost-effectively. 


The Department has made progress since our 1993 audit, mainly in 
managing the outside lawyers or law firms (“legal agents”) it hires for 
prosecutions (which are now the responsibility of the Public 
Prosecution Service of Canada, a new organization separate from 
Justice Canada). However, we believe the Department would benefit 
from having a senior executive position with authority for management 
practices and the administration of legal services. 
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Acquisition and travel credit cards can be a convenient and efficient 
way for the federal government to obtain and pay for goods and 
services, and many federal departments are encouraging their use. 
With spending power placed directly in the hands of thousands of 
government employees with numerous operational needs, it is 
important that the use of these cards be properly managed and the 
associated risks taken into account. Acquisition card spending rose 
from $350 million in 1999 to over $626 million in 2005, and travel 
card spending rose from $170 million to about $200 million. 


We examined the acquisition and travel card programs in National 
Defence, Fisheries and Oceans Canada, and Health Canada—the 
three departments that together account for about half of the total 
acquisition card use and a large portion of the travel card use in the 
government. We are pleased to report that our audit identified no cases 
of loss or abuse. 


We found that all three departments have adequate controls over the 
use of travel cards. For acquisition cards, the control framework relies 
significantly on post-payment verification and certification to ensure 
that transactions are properly authorized. 


We found that these key controls are not applied consistently and 
rigorously among the three departments or within each department. 
Lapses in the control and review of transactions leave departments 
exposed to risk. Moreover, post-payment verification and certification 
are not merely administrative niceties—they are an important 
application of law (section 34 of the Financial Administration Act). 


We also found that government-wide policies on the use of acquisition 
cards have not been updated to reflect the considerable evolution and 


increase in their use over the last decade. For example, the cards are 


now used for very high-dollar purchases, and some cards have high 
credit limits—one as high as $1.5 million at one of the departments we 
examined. Also, both National Defence and Fisheries and Oceans 
Canada issue acquisition cards in the names of vessels or 
organizational units and not in the names of specific individuals. While 
this practice is not consistent with the government's present policy, 
departments have said it is practical from an operational point of view. 
Their position might be reasonable, but they have not received the 
Treasury Board’s formal approval to issue the cards in this way. Clearly, 
a change in practice or a change in policy is in order. 
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Conclusion 


I hope that parliamentarians find the information in this report useful 
in holding the government to account for its stewardship of public 
funds and its delivery of services to Canadians. 


10 
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Chapter 1 


What we examined 


Why it’s important 


What we found 


Use of Acquisition and Travel Cards 


Main Points 


Acquisition cards are essentially credit cards that enable departments 
to purchase and pay for goods and services. The designated travel card 
is a credit card that government employees use for expenses such as 
hotels, car rental, and meals while they are travelling on business. We 
examined the acquisition and travel card programs in National 
Defence, Fisheries and Oceans Canada, and Health Canada, the three 
departments that together account for about half of the total 
acquisition card use and a large portion of the travel card use in the 
government. We also looked at the roles that the Treasury Board 
Secretariat (TBS) and Public Works and Government Services Canada 
(PWGSC) play in providing overall direction and guidance for these 
programs. 


Acquisition and travel cards can be a convenient and efficient way for 
the federal government to obtain and pay for goods and services, if it 
properly manages the associated risks. The use of acquisition and 
travel cards by federal government departments is growing, and many 
departments are encouraging their use to pay for goods and services 
and travel-related expenditures. Between 1999 and the end of 2005, 
acquisition card spending increased from more than $350 million to 
more than $600 million, and travel card spending from about 

$170 million to approximately $200 million. 


Periodically some misuse of acquisition or travel cards does occur in 
the government, receiving significant publicity and raising doubt about 
the adequacy of the controls in place. It is important that 
parliamentarians and Canadians be assured that public resources are 
not misused or abused by those who have access to convenient 
methods of paying for goods and services. 


- We identified no losses or abuse in our audit sample of acquisition 
card transactions. A framework of controls is in place to reduce the 
risks that acquisition cards are used improperly or without 
authorization. The control framework relies on post-payment 
verification and certification under section 34 of the Financial 
Administration Act (FAA) as the key control over the risks associated 
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with the cards. However, it and other key controls and safeguards 
required by the FAA are not applied rigorously by departments. 
Lapses in the control and review of transactions leave departments 
exposed to risk. 


The use of acquisition cards has evolved and increased considerably 
since 1998, when the current Treasury Board policy came into effect. 
For example, at National Defence and Fisheries and Oceans Canada, 
cards are now being used for very high-dollar purchases, and some 
cards have high credit limits—one as high as $1.5 million. 
Departments are encouraging their employees to make even more 
use of the cards. Departments differ in how they interpret the policy 
and how they use the cards, while the Treasury Board Secretariat has 
not updated the policy to take into account the changes in practice. 


In both National Defence and Fisheries and Oceans Canada, 
acquisition cards are sometimes in the names of vessels or 
organizational units, rather than in the names of specific individuals. 
While this use is not consistent with the present Treasury Board 
policy, departments have said it has operational merit. Although 
their position might be reasonable, they have not received the 
Treasury Board’s formal approval to use the cards in this way. 


The control framework for designated travel cards is adequate, given 
that the primary risk to the government is that it assumes the 
cardholder’s personal liability only after 90 days from the original 
statement date, and the government can manage the risk of default 
by using salary-offset provisions. However, the Treasury Board 
Secretariat and Public Works and Government Services Canada 
have not updated the travel card policies and procedures to reflect 
the provisions in the contract with the current service provider, 
stating that it is the government and not the cardholder that is 
responsible for unpaid balances after 90 days. 


The departments have responded. The departments are in general 
agreement with our recommendations. Their detailed responses follow 
each recommendation throughout the chapter. 
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What we examined 


Why it’s important 


What we found 


Federal Loans and Grants for 
Post-Secondary Education 


Human Resources and Social 
Development Canada and Canada 
Millennium Scholarship Foundation 


Main Points 


The Canada Student Loans Program of Human Resources and Social 
Development Canada (HRSDC) provides loans, Canada Access 
Grants, and Canada Study Grants to qualified students who need 
financial assistance for post-secondary education. The grants and the 
Millennium Bursary and Access Bursary programs of the Canada 
Millennium Scholarship Foundation are designed to help limit the 
amount of debt that recipients accumulate. 


We examined whether the Department and the Foundation have 
appropriate controls to ensure that they manage their responsibilities 
with due care and in compliance with legislation and other 
requirements. We looked at the controls that the Canada Revenue 
Agency has in place to ensure that the collection of defaulted Canada 
Student Loans complies with requirements. We also examined whether 
the Department and the Foundation ensure that information on the 
programs is easy to find and understand and that services are accessible 
and convenient. We examined how they measure their programs’ 
performance and report results. We did not examine the activities of 
the other partners in delivering the loan, grant, and bursary 
programs—the provinces and territories, and two private sector service 
providers contracted by HRSDC to handle most day-to-day operations 
of the Canada Student Loans Program. 


The federal government has said that Canada’s international 
competitiveness depends on having an educated and skilled labour 
force. To improve access to higher education is the main objective of 
both the Canada Student Loans Program and the Foundation. To this 
end, HRSDC loaned $1.9 billion to some 350,000 post-secondary 
students in 2005—06 and awarded 78,500 grants amounting to about 
$129 million. In 2006 the Foundation awarded about $298 million in 
Millennium Bursaries to some 100,000 students and over $40 million 
in Access Bursaries to more than 20,000 students. 


+ With a few exceptions, the Canada Student Loans Program is 
well managed. The Department has the necessary controls in place 
for disbursing loans and grants; it also obtains reasonable assurance 
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that its partners in delivering the Program are fulfilling their 
responsibilities. However, it does not have a clear strategy for 
measuring whether the Program has improved access to 
post-secondary education. Further, the Department has postponed 
completing its evaluation of the Program from 2006 to 2011. 


- The Foundation’s bursary programs are well managed. The 
Foundation has the necessary controls in place to ensure that 
eligibility for bursaries is assessed correctly and that payments are 
issued in the right amounts to the right people. It also obtains 
reasonable assurance that the provinces and territories are fulfilling 
their responsibilities in delivering the programs. The Foundation 
collects adequate information to report its achievements against 
objectives, and it meets its obligation to report the results of its 
activities to Parliament. 


The Department and the Foundation are taking steps to make 
prospective post-secondary students and their families more aware of 
the financial assistance available to them. The Department has also 
improved its communication to students about measures available to 
help manage their debts. 


The Canada Revenue Agency monitors the performance of private 
collection agencies, and Human Resources and Social Development 
Canada monitors the consistency of the information the agencies 
provide. However, neither organization adequately monitors the 
agencies’ compliance with key collection requirements of the 
Canada Student Loans Program. The Canada Revenue Agency has 
contracted with private collection agencies to handle about 

$450 million of the $800 million in defaulted loans owed to HRSDC 
(at 31 March 2006); it handles the remaining $350 million itself. 


* HRSDC and the Foundation have developed a good research 
relationship with one another. However, they do not have adequate 
arrangements for consulting each other to achieve their shared 
objective of improved access to post-secondary education. 


Human Resources and Social Development Canada, the Canada 
Revenue Agency, and the Canada Millennium Scholarship 
Foundation have responded. The Department, Agency, and 
Foundation have agreed with each of our recommendations and have 
committed to taking action. Their responses are included throughout 
the chapter. 
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Chapter 3 


What we examined 


Why it’s important 


What we found 


Human Resources Management 
Foreign Affairs and International Trade 


Canada 


Main Points 


The Department of Foreign Affairs and International Trade 
operates in a highly challenging and complex environment that 
requires it to respond quickly to changing local and international 
circumstances. It has staff based in Canada and in over 170 missions 
abroad, where it also employs a culturally diverse local workforce of 
over 5,000 employees who are subject to various local labour laws. 
In the 2005-06 fiscal year, the Department spent $830 million 

on salaries and benefits. 


We examined whether the Department plans for and manages its 
human resources in a way to ensure that it has the people with the 
right skills and competencies to carry out its mandate. Our audit 
included interviews, file reviews, and data analysis at the Department’s 
headquarters in Ottawa and at 12 of its missions abroad. 


Canada relies on the people who work in its foreign missions to be its 
“face abroad.” They represent Canada in such fields as political, trade 
and business relations, diplomacy, culture, and human rights. 
Canadians travelling abroad depend on the missions for a range of 
assistance, from replacing lost passports to determining their 
whereabouts, and seeing to their safety in emergency situations. The 
government relies on the Department for timely, high-quality advice 
on international issues to help it formulate foreign policy. The 
Department also has a position of leadership toward other government 
departments operating abroad to ensure a unified Canadian approach. 


It is important to Canadians that the Department have the right 
people in the right place at the right time and that it be able to act 
quickly and decisively in a rapidly changing environment. Given the 
complexity of its workforce, its international role, and the demographic 
challenge it faces, strategic planning and management of its human 
resources are critical to achieving its mandate. 


« The Department has no strategic human resources plan. It does not 
have a complete picture of the people, competencies, and experience 
it will need in the future, and it lacks basic information needed to 
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plan for and manage its workforce. For example, it does not have 
accurate information on the number of vacant positions it has to fill. 
Until recently, the Department has paid little attention to human 
resources planning. Since the 2005-06 fiscal year, it has engaged in a 
major effort to develop a strategic planning framework for managing 
its people, but much remains to be done. 


The Department has been unable through its recruitment, 
promotion, and assignment processes to fill its positions on a timely 
basis with enough people who have all the required skills and 
competencies. Fifty-eight per cent of its employees in the 
management category will be eligible to retire by 2010; for all 
categories combined, the figure is 26 percent. Already, departures 
now outnumber new hires. These are strong indications that the 
Department is at risk of not having the human resources it will need 
to carry out its mandate effectively. 


The process in place to compensate employees for living conditions 
abroad and provide incentives for hardship postings has not allowed 
the Department to respond in a timely way to changing 
circumstances and the problems employees face as a result. This 
hinders the Department’s ability to assign qualified staff to some of 
its missions around the world. Despite concerns expressed over many 
years by our Office and various stakeholders, this issue has not been 
resolved. 


The management of locally engaged staff, who constitute half of the 
Department’s workforce, gets little attention. The Department does 
not have all the information it needs to properly manage these 
employees and is not adequately supporting its missions in their 
human resources management responsibilities. 


The Department of Foreign Affairs and International Trade and the 
Treasury Board Secretariat have responded. The Department of 
Foreign Affairs and International Trade and the Treasury Board 
Secretariat have agreed with our recommendations. Their detailed 
responses follow the recommendations throughout the chapter. 
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What we examined 


Why it’s important 


What we found 


Canadian Agricultural 
Income Stabilization 


Agriculture and Agri-Food Canada 


Main Points 


The Canadian Agricultural Income Stabilization (CAIS) program is a 
federal-provincial-territorial program whose objective is to protect 
agricultural producers from drops in income due to circumstances 
beyond their control. The program is administered by Agriculture and 
Agri-Food Canada for all of Canada except Alberta, Ontario, Quebec, 
and Prince Edward Island, where the provincial governments 
administer the program. We did not audit the provinces’ delivery of the 
CAIS program. 


We conducted this audit at the request of the House of Commons 
Standing Committee on Agriculture and Agri-Food. We examined 
how the Department processes applications for income support, 
ensures that all parties are respecting the various monitoring provisions 
set out in the federal-provincial-territorial agreements, and measures 
and reports its performance to Parliament. 


The federal government has provided income support programs to 
farmers for decades. Agriculture and Agri-Food Canada receives about 
55,000 applications to CAIS each year. In the 2005-06 fiscal year, the 
Department spent about $1.1 billion Canada-wide, making the CAIS 
program the Department’s principal ongoing program of farm income 
support. 


* The CAIS program is very complex, and the Department does not 
make it transparent enough to producers; some have complained 
that they do not understand how the Department calculates their 
benefits. For example, when the Department changes financial or 
other information that producers submit, it does not explain the 
impact on the benefit amount. Consequently, producers do not 
receive, in an easy to understand manner, all the information they 
need to decide whether to ask for a payment adjustment or appeal an 
unfavourable decision. Further, there are long delays before 
producers are told whether they will receive a benefit and in what 


amount. 
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- The Department focuses on preventing and detecting overpayments 
made to producers, and the onus is on producers to question 
incorrect figures in the Calculation of Program Benefits statement 
sent by the Department. Improving transparency would make 
program delivery more consistent with the program’s objective, 
as set out in the Farm Income Protection Act—to protect the income 
of producers. 


Payment errors have been high, although they are beginning to 
decline. Staff who calculate payments work to standards that 
emphasize the number of applications processed and adherence to 
procedures more than accuracy of payments. While the Department 
collects data on the amounts and nature of the errors, it does not 
systematically use that information to improve its management of 


the program. 


« Some employees who processed applications also worked as private 
consultants to help producers prepare their applications. We are 
concerned that such conflicts of interest could provide an unfair 
financial advantage to some applicants. 


* Some of these problems are not new; our past audits of previous farm 
income support programs have found similar problems. 


The Department has responded. In its response to each 
recommendation throughout the chapter, Agriculture and Agri-Food 
Canada has indicated the actions it has taken, will take, or will 
consider taking. 


20 


Chapter 4 


Report of the Auditor General of Canada—May 2007 


Chapter 5 


What we examined 


Why it’s important 


Managing the Delivery of Legal 
Services to Government 


Department of Justice Canada 


Main Points 


Each department and agency of the federal government requires legal 
counsel to provide it with legal advice, to draft regulations and 
legislation, and to represent it in court. The Department of Justice 
Canada (Justice Canada) provides these services to all departments 
and agencies, with a few exceptions. In addition, it enforces regulations 
and legislation through prosecution, and examines the need for 
changes in legislation. We examined whether the Department 
effectively manages the delivery of legal services to meet the needs of 
government. We also looked at whether it takes appropriate steps to 
ensure the quality of the legal services and whether it delivers them in 
a cost-effective manner. 


Up to the end of our examination, prosecution services were provided 
by the Federal Prosecution Service—a branch of Justice Canada—and 
were included in our audit scope. Prosecution services became the 
responsibility of the Public Prosecution Service of Canada when it was 
created under the Federal Accountability Act, which received royal 
assent on 12 December 2006. This new organization is separate from 
Justice Canada and reports to the Attorney General. Our findings and 
recommendations relate to the work now carried out by the Public 
Prosecution Service of Canada as well as the work of Justice Canada. 


Canadians rely on lawyers to provide them with advice on, for 
example, purchasing or selling property, entering contracts, or settling 
disputes. Government departments and agencies rely on their lawyers 
for the same services. Justice Canada also covers a range of legal 
advisory, litigation (including prosecution), and legislative services. 
The Department’s job is complicated by the fact that its lawyers 
represent the interests not only of each client department but also of 
the government as a whole. 


Approximately 2,500 lawyers work for Justice Canada. Since our last 
audit in 1993 the annual cost of the Department’s legal services has 
more than tripled to over $600 million. This significant increase is 
attributed to growing complexity and volume of litigation and 
prosecutions, as well as the growth in demand for legal services to 
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What we found 


departments since the introduction of the 1982 Charter of Rights and 
Freedoms. The Department has also faced more demand for its services 
in areas such as Aboriginal affairs, taxation, drug prosecutions, and 
immigration. 


- The Department puts considerable effort into ensuring the quality of 


the legal services it provides to the government—its stated objective. 
While it does not have an overall quality management system, it uses 
several practices to promote quality. To date, its quality management 
efforts have focused on complex and high-risk litigation. It has also 
invested in information systems, development of guide books, and 
training and development of its staff. However, the Department has 
not expressed what it means by quality. This impedes its ability to 
deliver consistent quality of service to its 42 client departments and 
assess whether it is meeting its objective. It also lacks a system to 
provide senior management with ongoing assurance that all services 
meet established minimum standards. 


The Department lacks information on the volume of its work and its 
use of staff—information that would demonstrate whether it is 
delivering legal services cost-effectively. It has some time-keeping 
data for some lawyers, but recording time worked has only recently 
become a department-wide requirement. Justice Canada does not 
collect and use department-wide financial and workload information 
for monitoring trends, planning, setting priorities, managing 
resources, analyzing service delivery options, or identifying 
opportunities to control costs. 


* Justice Canada shares the cost of legal services with departments. It 


has numerous financial arrangements with client departments— 
more than 100 arrangements with the eight departments we 
interviewed. Its financial arrangements are inconsistent, poorly 
documented, and inefficient to administer. Justice Canada has little 
incentive to control the costs of its services, because any 
unanticipated costs of legal services are borne by the client 
departments involved. Client departments also have little incentive 
to control their legal costs, which they are able to absorb as a small 
percentage of their total costs; for significant costs they can seek 
additional funding through the Treasury Board. Without good 
information on costs, it is difficult to manage and control the 
growing demand in departments for legal services and to find more 
efficient ways of providing them. 
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The total costs of legal services incurred across the government are 
not consolidated for reporting to Parliament. Justice Canada reports 
the total costs of its legal services and the total amount recovered 
from departments. Each client department might report the costs of 
the legal services provided to it by Justice Canada, but other legal 
costs paid by departments are not readily identifiable because they 
are not reported separately from other operating expenditures. 


Much of what we found had been identified in our 1993 audit as 
weaknesses. The Department has made progress in managing legal 
risk, managing legal agents, and introducing time-keeping into the 
Department. However, we found no evidence of results in some 
important areas, despite efforts by the Department—notably the use 
of department-wide information on workload and human resources 
and its financial arrangements with client departments. 


The Department has responded. The Department of Justice Canada 
agrees with each of our recommendations. Its detailed response follows 
each recommendation throughout the chapter. 
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Modernizing the NORAD System in 
Canada 


National Defence 


Chapter 6 Main Points 


What we examined § Canada and the United States are partners in the air defence of North 
America under the North American Aerospace Defense (NORAD) 
agreement. Under this agreement, Canada and the US operate an 
integrated and interoperable air surveillance and control system to 
secure North American airspace. In 1997, the two countries agreed to 
upgrade and modernize this system. Originally, the Canadian 
government approved funding for about $93 million to modernize the 
Canadian portion of the NORAD system—the Canadian Air Defence 
Sector Air Operations Centre. Since the original approval for 
modernization, the project has undergone several changes and cost 
increases. We examined how the modernization has progressed, what it 
has cost National Defence, and what has been delivered. 


Why it’s important Heightened security demands since September 11, 2001, have put 
emphasis on the need to modernize the NORAD surveillance and 
control systems. National Defence has a significant investment in the 
modernization project and needs to implement it in a way that is 
cost-effective and ensures that operational requirements are met. In 
May 2006, Canada and the United States renewed the NORAD 


agreement to continue operations in both countries. 


What we found = + Modernization was originally expected to cost National Defence 
$87 million, plus about $6 million for a definition phase, and was to 
be completed by 2001. However, we estimate that the Department 
has spent about $125 million so far, and bringing the modernization 
to completion is expected to cost another $18 million. Other related 
expenses of about $13 million will bring the expected total cost to 
about $156 million. The project took longer than planned to deliver 
and has cost more than initially expected because of problems with 
system development and project management. Recently, the 
Department successfully installed a new air surveillance and control 
system at its sector air operations centre, and it reports that the 
system is meeting initial operating requirements. However, 
anticipated savings of up to $16 million a year in personnel and 
operating costs have not materialized. 
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Neither National Defence nor the government made appropriate use 
of mechanisms available for managing large, high-risk projects like 
this one. Early signs that the project was in trouble and that costs 
were climbing did not prompt moves to strengthen its oversight. We 
asked why, for example, a decision was not made at the time to 
designate this a Major Crown Project. Treasury Board Secretariat 
officials initially declined to provide us access to the working papers 
on this issue because they viewed these as cabinet confidences. At 
the conclusion of the audit and after a search, Treasury Board 
Secretariat officials assured us that there were in fact no Treasury 
Board Secretariat working papers to provide to us. 


Information provided to the government for increased project 
funding was not always timely or accurate, and did not always reflect 
the risks of the project or how its scope had changed. 


National Defence entered into an agreement with the United States 
Department of Defense as an equal partner, during the development 
phase of the first effort to implement a new air surveillance and 
control system, and on that basis it agreed to pay 50 percent of 
common costs. Yet it could not provide us with evidence to 
demonstrate that it had in fact shared control of the project. Later, 
problems appeared in the Department’s ability to control its own 
development of a second, replacement air surveillance and control 
system, which it was working on for the Canadian Air Defence 
Sector. Work on this second system was stopped in favour of a third 
system developed for NORAD, mainly with US Department of 
Defense input into the requirements. This air surveillance and 
control system has now been installed and is operating at the 
Canadian Air Defence Sector Air Operations Centre. Future 
modernization and upgrade work is planned, but National Defence 
still has not clearly set out its own Statement of Operational 
Requirement and needs to review requirements before continuing. 


National Defence has responded. The Department agrees with our 
recommendations and is proposing actions to address concerns. 
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What we examined The Forensic Laboratory Services (FLS) of the Royal Canadian 
Mounted Police (RCMP) provides forensic analysis services for 
criminal cases to Canada’s law enforcement community, which 
includes Canadian police agencies, Crown counsel, and other federal, 
provincial, and municipal agencies. It operates laboratories in six cities 
across Canada. . 


In 2005, the House of Commons Standing Committee on Justice and 
Human Rights, Public Safety and Emergency Preparedness heard 
conflicting testimony on the performance of the Forensic Laboratory 
Services. The Committee subsequently asked the Auditor General of 
Canada to audit the performance of the FLS, and the status of DNA 
cases and service requests. We examined the timeliness of service 
delivery by the FLS, the quality management system, consultation with 
clients, and performance reporting to Parliament. We did not audit the 
quality of the forensic science and offer no opinion, positive or 
negative, on the validity of scientific analysis used. 


Why it’s important © The RCMP’s Forensic Laboratory Services is the main provider of 
forensic analysis for most police agencies and courts in Canada. 
Forensic labs are an important element of the criminal justice system. 
Investigators and prosecutors rely on forensics to help identify or 
eliminate suspects and to provide evidence that can withstand scrutiny 
in court. Delays in obtaining results of forensic analysis can slow police 
investigations and leave criminals on the street to reoffend. 


What we found = + For the most part the FLS does not meet its own turnaround targets 
for completing service requests. Although it can process urgent 
service requests in less than 15 days, they account for only | percent 
of all service requests. In the remaining 99 percent categorized as 
routine, the FLS is unable for the most part to meet the 30-day target 
it has set for them. While average turnaround times have improved 
for all other types of analysis, for DNA analysis requests they have 
worsened—from 91 days in 2003-04 to 114 days in 2005—O06— 
despite increased spending and additional staff. The backlog of DNA 
requests is a major contributor to the long turnaround times. 


26 Chapter 7 Report of the Auditor General of Canada—May 2007 


MAIN PoINTS—CHAPTERS 1 TO 7 


Although the labs now have a national quality management system 
in place, in practice there are significant weaknesses in how the FLS 
defines, records, monitors, and resolves quality issues (situations 
where there is a concern that the validity of forensic work is 
unreliable for any reason), including those related to lab results. 
Furthermore, the national quality management system failed to 
identify problems with the new automated process for DNA analysis. 
At the conclusion of our audit work, the FLS quality management 
system was not functioning as designed and could not provide 
assurance of quality to senior management. 


The RCMP does not give clients—police forces and prosecutors— 
adequate opportunity to influence how the FLS operates. For 
example, clients told us that although the RCMP conducts client 
visits, it consults them very little on their needs regarding matters 
such as changes to the labs’ services, priorities, and service standards. 
Clients have little opportunity to negotiate turnaround times for 
service requests, and a recent change in policy gives them little say in 
the number of exhibits they are allowed to submit—no more than 
eight with each request. 


Although its new Laboratory Information Management System 
enables it to examine the performance of the FLS, the RCMP is not 
keeping its commitment to report to Parliament on performance; nor 
is it reporting to clients on FLS performance. 


The RCMP has responded. The RCMP agreed to our 


recommendations and is preparing action plans to address them. 
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Use of Acquisition and Travel Cards 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Acquisition cards are essentially credit cards that enable departments 
to purchase and pay for goods and services. The designated travel card 
is a credit card that government employees use for expenses such as 
hotels, car rental, and meals while they are travelling on business. We 
examined the acquisition and travel card programs in National 
Defence, Fisheries and Oceans Canada, and Health Canada, the three 
departments that together account for about half of the total 
acquisition card use and a large portion of the travel card use in the 
government. We also looked at the roles that the Treasury Board 
Secretariat (TBS) and Public Works and Government Services Canada 
(PWGSC) play in providing overall direction and guidance for these 
programs. 


Acquisition and travel cards can be a convenient and efficient way for 
the federal government to obtain and pay for goods and services, if it 
properly manages the associated risks. The use of acquisition and 
travel cards by federal government departments is growing, and many 
departments are encouraging their use to pay for goods and services 
and travel-related expenditures. Between 1999 and the end of 2005, 
acquisition card spending increased from more than $350 million to 
more than $600 million, and travel card spending from about 

$170 million to approximately $200 million. 


Periodically some misuse of acquisition or travel cards does occur in 
the government, receiving significant publicity and raising doubt about 
the adequacy of the controls in place. It is important that 
parliamentarians and Canadians be assured that public resources are 
not misused or abused by those who have access to convenient 


methods of paying for goods and services. 


- We identified no losses or abuse in our audit sample of acquisition 
card transactions. A framework of controls is in place to reduce the 
risks that acquisition cards are used improperly or without 
authorization. The control framework relies on post-payment 
verification and certification under section 34 of the Financial 
Administration Act (FAA) as the key control over the risks associated 
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with the cards. However, it and other key controls and safeguards 
required by the FAA are not applied rigorously by departments. 
Lapses in the control and review of transactions leave departments 
exposed to risk. 


The use of acquisition cards has evolved and increased considerably 
since 1998, when the current Treasury Board policy came into effect. 
For example, at National Defence and Fisheries and Oceans Canada, 
cards are now being used for very high-dollar purchases, and some 
cards have high credit limits—one as high as $1.5 million. 
Departments are encouraging their employees to make even more 
use of the cards. Departments differ in how they interpret the policy 
and how they use the cards, while the Treasury Board Secretariat has 
not updated the policy to take into account the changes in practice. 


In both National Defence and Fisheries and Oceans Canada, 
acquisition cards are sometimes in the names of vessels or 
organizational units, rather than in the names of specific individuals. 
While this use is not consistent with the present Treasury Board 
policy, departments have said it has operational merit. Although 
their position might be reasonable, they have not received the 
Treasury Board’s formal approval to use the cards in this way. 


The control framework for designated travel cards is adequate, given 
that the primary risk to the government is that it assumes the 
cardholder’s personal liability only after 90 days from the original 
statement date, and the government can manage the risk of default 
by using salary-offset provisions. However, the Treasury Board 
Secretariat and Public Works and Government Services Canada 
have not updated the travel card policies and procedures to reflect 
the provisions in the contract with the current service provider, 
stating that it is the government and not the cardholder that is 
responsible for unpaid balances after 90 days. 


The departments have responded. The departments are in general 
agreement with our recommendations. Their detailed responses follow 
each recommendation throughout the chapter. 
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Introduction 


What are acquisition cards, and how are they managed? 


11 Acquisition cards are credit cards that enable departments to 
purchase and pay for goods and services. They were initially approved 
for use in government departments and agencies in 1991 and are now 
offered through contracts with two Canadian banks. The Bank of 
Montreal (BMO) provides MasterCard credit card services, and the 
Canadian Imperial Bank of Commerce (CIBC) provides Visa credit 
card services. Departments can use either or both of these acquisition 
card services, depending on their operational needs. 


1.2 The contracts with the service providers offer a rebate program, 
payable directly to the departments. Rebates are based on variables 
such as the level of expenditures on the cards and how quickly 
departments pay the balance owing. 


13 Since their introduction, the use of acquisition cards has 
increased significantly. The acquisition card program started in 1991, 
and by 1992 there were approximately 2,000 cards in use. In 1996, the 
period examined in our first audit on acquisition cards, the government 
had approximately 20,000 cards, and transactions were valued at about 
$170 million. As of December 2005, approximately 35,600 cards were 


in use, with transactions exceeding $600 million (Exhibit 1.1). 


14 Until 1 September 2006, the Treasury Board Secretariat (TBS) 
was responsible for contracts with the service providers, policy, and 


Exhibit 1.1 The number of acquisition cards used in the federal government increased substantially 
from 1992 to 2005 


Number of acquisition cards 
(Thousands) 


1992 1996 2005 
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oversight of the operations of the acquisition card program. After this 
date, Public Works and Government Services Canada (PWGSC) 
assumed responsibility for contracts with the service providers and 
program oversight. The Treasury Board is still responsible for 
government-wide policy regarding acquisition cards. 


1.5 . Departments are responsible for establishing specific 
departmental policies and procedures to ensure the economical and 
efficient use of acquisition cards, and for designating a departmental 
coordinator. Each departmental coordinator is responsible for the 
following: 


¢ authorizing and issuing acquisition cards, 


* monitoring program design to ensure reliable controls exist over 
the use of cards, and 


* managing their department’s card programs. 


Finally, departments are also responsible for carrying out reviews and 
audits to determine whether cards are being used according 
to Treasury Board policy. 


What are travel cards, and how are they managed? 


16 The “designated travel card” is a credit card that individual 
government employees use to pay expenses such as hotel, car rental, 
and meals while they are travelling on business. Employees are 
responsible for paying their own travel card balances and are 
reimbursed for these business expenses, once they complete a travel 
claim. Staff who arrange travel for each responsibility centre also use 
the “designated-responsibility-centre travel card” to pay airfares and 
travel agency fees. 


17. Through a subcontract with Accenture, American 

Express (AMEX) is the travel card service provider. The government 
earns rebates, based primarily on the total expenditures relating to 
both the designated travel cards and the designated-responsibility- 
centre travel cards. 


18 In 2005 government employees used designated travel cards to 
make approximately $200 million in purchases; this was an increase 


from about $170 million in 1999. 


19 =PWGSC has overall responsibility for managing the travel card 
program. Departments are responsible for authorizing, issuing, 
monitoring, and cancelling the use of travel cards. Both employees and 
the department play an important role in managing the travel card 
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program. If an employee fails to pay the outstanding card balance after 
90 days, the liability for this outstanding balance is transferred to the 
government. In such cases, the department must then recover the 
money from the employee. 


Focus of the audit 


1.10 The audit focused on three departments: Fisheries and 
Oceans Canada, Health Canada, and National Defence. These 
three departments are among the largest users of travel cards and 
account for about half of the government’s expenditures on 
acquisition cards. 


1.11 We looked at whether the management control framework over 
the use of the acquisition cards and designated travel cards (travel 
cards) is appropriate and whether the government is aware of the 
extent of the intended benefits realized from these programs. We did 
not include as part of this audit the management of the 
designated-responsibility-centre travel cards that are used by 
responsibility centres to pay for airfares and travel agency fees. 


112 Our audit focused on the processes for managing the card 
programs in the three departments—not on the merits of individual 
expenditures. We also assessed the roles of the Treasury Board 
Secretariat and Public Works and Government Services Canada in 
providing overall direction and guidance on these programs. 


113 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


1.14 Appropriate policies from the Treasury Board are critical to 
delivering any program. No less important are appropriate procedures 
and guidelines at the departmental or operational level. This is 
particularly true for acquisition and travel card purchasing, where 
spending power is placed directly in the hands of thousands of 


government employees with numerous operational needs. 


Government-wide policies need to be updated 


1.15 In 1998, the Treasury Board revised its Policy on Acquisition 
Cards. This policy emphasized the need for additional controls at the 


departmental level for improved monitoring and analysis, and for the 
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Delegated financial authority —Occurs when 
government authorizes a specific individual to 
initiate or approve financial transactions, within 
established limits. 


use of information supplied by the service providers in an electronic 
format. The Treasury Board Secretariat also issued the Acquisition 
Cards Program—Best Practices Guide in 1998 and the Acquisition Cards 
Program—Management Guide in 1999. These documents contain useful 
information on the history, use, and benefits of acquisition cards. They 
also describe the acquisition card control framework, recommend 
practices for acquisition card use, and highlight internal controls that 
will assist managers in monitoring acquisition card activities. 


1.16 There have been no updates to the Treasury Board Policy on 
Acquisition Cards or either of the guides since they were issued. We 
note that the Treasury Board Secretariat’s Acquisition Cards Program— 
Best Practices Guide refers to a future version of the guide that would 
cover topics such as best practices and provide advice on types of 
purchases, card limits, and delegated financial authorities. With the 
changing environment in the use of acquisition cards, namely the 
increase in the number of cardholders, increase in total dollars spent 
on the cards, and the use of the cards for high-dollar transactions, it is 
important that control processes and guidance not become outdated. 
This could possibly expose the government to additional risks. The 
Treasury Board is currently updating its Policy on Acquisition Cards. 


1.17. At the departmental level, we found that the three departments 
examined had generally developed and issued specific acquisition card 
policies and directives consistent with the policy established by the 
Treasury Board. However, the departmental policies of both Fisheries 
and Oceans Canada and National Defence permit acquisition cards to 
be issued in the name of an operational unit or a vessel, rather than in 
the name of individuals. We found that such cards in fact exist. 
Although we can understand that these “corporate” cards provide 
flexibility, the current service-provider contracts make no specific 
reference to such cards, and the Treasury Board Policy on Acquisition 
Cards states that only the “person whose name appears on the card” 
can use it. 


118 We also found that the Health Canada acquisition card policy 
did not explicitly restrict cardholders from approving their own 
acquisition card expenditures and that such situations were, in fact, 
occurring. This policy and practice is, in our opinion, not consistent 
with the fundamental control principle that there be a division of 
duties between those executing a transaction and those approving the 
subsequent expenditures. As this audit was being completed, Health 
Canada was in the process of revising its departmental acquisition card 
policy to eliminate this practice. 
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119 Recommendation. The Treasury Board Secretariat should 
update the policy with respect to acquisition cards including whether 
“corporate” acquisition cards should be permitted. 


The Treasury Board Secretariat’s response. The Treasury Board 
Secretariat agrees with the recommendation. As part of the renewal of 
‘Treasury Board’s financial management policies, the Directive on 
Acquisition Cards is being updated. The revised directive will not 
permit the issuance of acquisition cards in the name of an organization 
or asset, such as a vessel, as it would have an impact on the 
government's ability to monitor and control the usage of the card as 
well as on the merchant's ability to accept the card as payment. 
Forecasted completion for the updated directive is 31 March 2008. 


The existing policy on travel cards does not reflect the terms of the current contract 
with the service provider 


1.20 The Treasury Board Policy on Using Travellers Cheques, Travel 
Cards and Travel Accounts, which is scheduled to be replaced with a 
Treasury Board directive on travel cards, governs the use of travel 
cards within the government. Public Works and Government Services 
Canada (PWGSC) has overall responsibility for managing the travel 
card program. In this respect, PWGSC has regular communications 
with AMEX and departments regarding the use of the designated 
travel card. 


1.21 The Treasury Board policy came into effect on | January 1998, 
and was amended in March 2000. We found that it does not reflect the 
terms of the current contract between the government and Accenture, 
with AMEX as the subcontractor. The contract states that full 
payment for unpaid balances on travel cards will be made by the 
government after 90 days, whereas the policy leaves payment 
responsibility with the cardholder. 


1.22 At the time of our audit, we found that only Fisheries and 
Oceans Canada had a travel card policy in place. Both Health Canada 
and National Defence indicated that they had draft travel card 


policies, which were being finalized. 


1.23 We noted that all three departments had effectively 
communicated information to travel cardholders about the issue of 
transferring the payment responsibility from the employee to the 
government, as stated in the latest contract with Accenture. 
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Quality assurance of account verification of 
acquisition card transactions—A practice 
that allows management to re-examine monthly 
transactions and to monitor compliance. During 
this re-examination some transactions and 
related documents are checked to make certain 
that the person with delegated authority has 
signed that the goods or services were provided 
and that the right price was paid. 


1.24 Recommendation. The Treasury Board Secretariat should 
complete and issue its directive on travel cards. 


The Treasury Board Secretariat’s response. The Treasury Board 
Secretariat agrees with the recommendation. As part of the renewal of 
Treasury Board’s financial management policies, the Directive on Travel 
Cards and Travellers Cheques is being updated. Forecasted completion 
for the updated directive is 31 March 2008. 


1.25 Recommendation. Public Works and Government Services 
Canada should provide departments and agencies with operational 
guidelines pertaining to the provisions of the contract with the travel 
card service provider, including guidelines pertaining to the 
responsibility of the government for unpaid travel card balances. 


Public Works and Government Services Canada’s response. Public 
Works and Government Services Canada agrees with this 
recommendation and will provide departments and agencies with 
operational guidelines pertaining to the provisions of the contract 
with the travel card service provider. 


Training practices for departmental coordinators and quality assurance personnel 
are lacking 


1.26 ‘Training is a fundamental requirement to effectively implement 
policies and procedures. We expected that there would be training 
practices in place to ensure that those responsible for the use, 
processing, and oversight of acquisition and travel cards would be fully 
aware of their roles and responsibilities. 


1.27. There is an absence of formal training for both the departmental 
and regional-level card coordinators in all three departments. 
Interviews with departmental card coordinators indicated that they 
had no specific training for their roles and responsibilities with the 
acquisition card program. We found evidence of informal on-the-job 
training being provided. 


1.28 In the three departments, we found no evidence of formal 
training for those persons responsible for the quality assurance of 
account verification of acquisition card transactions. Those 
responsible for quality assurance require a good knowledge of the 
documentation and financial system information associated with 
acquisition cards and financial processes. 


8 Chapter 1 


Report of the Auditor General of Canada—May 2007 


USE OF ACQUISITION AND TRAVEL CARDS 


1.29 Travel cards pose a lower level of risk, as the agreement between 
the cardholder and the government is clear and concise. In addition, 
the unpaid balances become a liability to the government only after 
90 days. Further, with the salary-offset provisions, the government has 
limited exposure to losses. For these reasons, we did not expect that 
formal training would be provided to travel cardholders, but rather 
that their roles and responsibilities would be effectively communicated. 


1.30 Good communication with cardholders. The results of our 
examination indicated that all three departments have good 
communication initiatives to ensure that the roles and responsibilities 
of cardholders for both acquisition and travel cards are documented 
and communicated. These initiatives include intranet references, 
policies, and procedures on the use of acquisition and travel cards; 
internal publications; and cardholder acknowledgement forms. 


1.31 Recommendation. Fisheries and Oceans Canada, Health 
Canada, and National Defence should formalize training for regional 
and departmental coordinators and quality assurance personnel to 
ensure that information about all issues related to acquisition and 
travel card management within a department is understood by those 
personnel, and that the information is disseminated to program 
managers. 


Fisheries and Oceans Canada’s response. Fisheries and Oceans 
Canada agrees. Over the next 12 months, the Department will 
implement a training program for card coordinators and quality 
assurance personnel, and will improve the departmental coordination 
of card management and of the quality assurance function. In addition, 
the Department will take the necessary steps to inform card 
coordinators of the availability on the website of training material and 


tools accessible to them. 


Health Canada’s response. Health Canada agrees that training 
should be available for these groups and will explore alternatives for 
the delivery of training in respect of departmental coordinators, 
including the possible provision of this training through central 


agencies or the Canada School of Public Service. 


National Defence’s response. National Defence agrees. The 
Department is developing a range of training packages for all personnel 
involved in the use of acquisition and travel cards. It is expected that 
the formal training program will be implemented through regional 


coordinators over the course of 2007. 
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Verification and monitoring of 
card transactions 


Account verification and certification— 


0 
e 
a 


Tk 
S| 


ccurs when a person reviews a transaction to 
isure that the goods or services were provided 
id that the price for the goods or services was 


the same as promised or expected (verification). 


nen a person with delegated financial authority 
gns the document to confirm that a review has 


deen performed (certification). Together, both of 


tI 


ese actions complete the requirement called 


“section 34 certification and verification.” 


Centralized payment of acquisition card transactions requires modifications to 
key controls 


1.32 Control frameworks are tools that help management in overseeing 
operations. An effective control framework incorporates controls that 
are appropriate for the benefits and risks of a given activity. 


1.33 The control framework for acquisition cards uses controls to both 
prevent and detect problems, to reduce the risks that cards are not 
properly authorized or used, and to ensure that all corrective actions 
needed are taken. Key controls for acquisition cards are primarily based 
on the Financial Administration Act (FAA) and associated Treasury 
Board policies. The FAA requires that fundamental controls for 
procuring and paying goods and services be respected. 


1.34 Under a traditional invoice the processes of procuring, paying, 
and verifying a purchase involve applying sections 32, 33, and 34 of 
the FAA. These three sections require the following: 


¢ that funds be available and committed for the purchase 
(section 32); 


* that no payment for a purchase be made unless it has been 
properly requisitioned (section 33); and 


¢ that these goods or services be performed, supplied, or rendered, 
and the price be as stated in the contract—verification—and that 
the person with delegated financial authority signs that the 
verification has been completed—certification—(section 34). 


All three steps usually take place when a purchase is formally 
requisitioned and approved before being made and before the invoice 
is paid. However, with acquisition card purchases in departments that 
use central billing, the money is paid before all these controls can be 
implemented. 


1.35 Each of the three departments covered by this audit has a central 
billing process in place to pay acquisition card statements. With central 
billing, each department receives a consolidated statement from the 
service provider that incorporates all the cardholder statements across 
the organization. This combined statement enables each department 
to make one monthly payment to the service provider. Furthermore, 
this practice enables a department to make payments quickly (usually 
within four days) to maximize any rebates available. Being able to act 
promptly is advantageous, as the size of the rebates to a department is 
determined by the volume of transactions and the speed with which 
that department pays its service provider. 
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1.36 When using acquisition cards in a centralized billing process, 
departmental staff need to modify the key controls so that they can 
expedite payments and 


* avoid interest charges, 
* maximize rebates, and 


* allow cardholders the flexibility to make purchases without 
requiring an excessive number of pre-approvals. 


Therefore, departmental staff need to replace the traditional means of 
applying FAA controls with the exercise of careful control in 
authorizing and issuing acquisition cards, selecting the credit limit for 
each card, and applying the Treasury Board Policy on Account 
Verification. 


137. The Treasury Board Policy on Account Verification outlines 
circumstances that allow account verifying and certifying to occur after 
the payment is made. The quality assurance requirements for the 
Policy state that those with payment authority must provide assurance 
that the process of verifying and certifying the purchase is adequate 
and is being properly and conscientiously followed. When this is the 
case, the Policy permits departments to apply the account verification 
required by section 34 after the central bill is paid, as long as the 
departments are satisfied that their preventive and detective controls 
are strict enough to reduce or eliminate errors in the payment of 
acquisition card transactions. 


The quality of the verification and certification processes over acquisition card 
transactions varies 


1.38 As part of our audit, we randomly selected 165 acquisition card 
transactions from the three departments during the 2005 calendar 
year. We tested whether these transactions had been properly verified 
and certified and that all were properly supported with invoices. We 
also tested whether there was proper authorization and approval of the 
acquisition card and whether cardholders acknowledged their roles 


and responsibilities (through completed acknowledgement forms). 


1.39 Our testing found that all three departments were able to 
provide appropriate supporting documentation for each transaction. 
With respect to acknowledgement forms completed by the 
cardholders, Health Canada could not provide these forms in 

38 percent of the cases, and similarly Fisheries and Oceans Canada was 
unable to provide the forms in 13 percent of the cases. In 99 percent of 


the cases, National Defence was able to provide these forms. 
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1.40 We found that the quality of verification and certification work 
varied widely among the three departments and within each 
department, as well. Specifically, the results of our testing indicated 
that the control framework is in place; however, it is not being applied 
consistently. The results of our examination indicate that the account 
verification and/or certification process (required by section 34) is not 
performed in a rigorous manner (Exhibit 1.2). We noted the following 
problems: 


* transactions that had been approved for payment were not 
subsequently verified and certified; 


* transactions were verified and certified by individuals that did not 
have the delegated authority; and 


* transactions were verified and certified by the cardholders who 
made the transactions, rather than by another authorized 
individual. 


Exhibit 1.2 The results of a statistical sample of acquisition card transactions highlight that account 
verification and/or certification is not rigorously applied 


National Fisheries and — 
Defence Oceans Canada | Health Canada 
Total number of transactions 391,023 218,783 139,926 
Sample size 60 60 45 
Transactions without appropriate 
section 34 certification and/or 
verification 45% 14% 21% 


141 Among the transactions without appropriate section 34 
certification and/or verification, our testing revealed that at Health 
Canada, all of the transactions involved cardholders who had 
section 34 authority and who had approved their own card 
transactions. In our opinion, this practice is not consistent with the 
fundamental control principle that there be a division of duties 
between those executing and those approving the transactions. For 
National Defence, among the transactions without appropriate 
section 34 certification and/or verification, our testing identified 

15 transactions where there was a signature under section 34. 
However, the department could not produce the signature cards to 
confirm that the person who was making the certification under 
section 34 had the delegated authority. 
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1.42 Our audit also found concerns with the quality assurance aspects 
of the section 34 verification and certification process, the 
establishment of credit limits, the card cancellation practices, and 
inventory control. Specifically, we found the following: 


¢ The quality assurance work was inconsistent among the different 
departments. In addition, we found that the number of 
transactions that quality assurance staff reviewed, the nature of 
their review, the number of staff assigned to the reviews, and the 
timeliness of the reviews were different within the various regions 
of the three departments. 


¢ Although Fisheries and Oceans Canada had established 
guidelines to assist in determining credit limits for acquisition 
cards, we found that, in some cases, the guidelines were not 
followed or credit limits were much greater than the expenditure 
pattern of the cardholder. In addition, we noted that only one of 
the four regions that we had visited had implemented a plan for a 
regular review of credit limits to ensure that the limits were in line 
with Fisheries and Oceans’ guidelines. At National Defence, we 
also noted that there were no established guidelines with respect 
to credit limits for acquisition cards, that some of the cards had 
credit limits much greater than the expenditure pattern of the 
cardholder, and that there was no regular review of these credit 
limits to ensure they were appropriate for the circumstances. 


¢ The procedures established for cancelling acquisition cards held 
by employees who had left a department were inconsistently 
applied in the three departments. For example, at Fisheries and 
Oceans Canada we found one case where seven employees 
continued to use the acquisition card of a former employee for 
government expenses, for one month before the card was 
cancelled. In addition, at National Defence we found a variation 
in procedures between military and civilian cardholders. We found 
strong procedures governing the cancellation of cards held by 
military personnel; however, departmental coordinators could not 
assure us that cards for civilian personnel were cancelled in a 


timely manner. 


¢ When purchases met a department's threshold for being 
recognized as inventory items, they were not being consistently 
recorded in the departmental inventory management systems. 
Specifically, at National Defence we found that four of eight 
items were not recorded in the inventory management system, 


although we were able to physically locate all eight items. 
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1.43 Our audit also found that the acquisition card had been used for 
the payment of some very large transactions such as refuelling costs. 
Departments stated that such transactions fell within their delegated 
payment authority for acquisition card purchases and provided 
convenience in meeting operational needs. However, we found that 
these transactions represented a large percentage of the dollar value of 
transactions undertaken at Fisheries and Oceans Canada and National 
Defence (Exhibit 1.3). 


144. As previously outlined, the control framework for acquisition 
cards effectively relies only on post-payment verification to ensure the 
appropriate payment of goods and services. As a result we identified 
high-dollar-value transactions as possibly being subject to a higher 
degree of risk of error. Therefore, we performed additional audit work 
on 21 of these transactions. 


145 The results of our examination of these high-dollar-value 
transactions also identified failures to properly apply the post-payment 
verification controls. Specifically, the results of our testing indicated 
that six of these transactions did not have appropriate section 34 
approval. It is our opinion that these high-dollar-value transactions are 
larger than the typical transactions that the Treasury Board Secretariat 
(TBS) had in mind when it developed the acquisition card program. 
For this reason, departments using the acquisition card for these 
transactions, need to ensure that they consider the inherent risk 
associated with these transactions and rigorously apply the 
post-payment verification procedures. The TBS must also ensure, as 
previously stated, that its policies and procedures are updated to 


Exhibit 1.3 Acquisition card transactions of amounts larger than $5,000 represent a large percentage of total dollars spent within two departments 


Fisheries and 


National Defence Oceans Canada Health Canada 
Total number of transactions 391,023 218,783 139,926 
Total number of transactions larger than $5,000 2,509 1,093 33 
Percentage of total number of transactions 0.64% 0.50% 0.02% 


a= = ee ee ny 
Total value of all transactions $165,653,405 $ 87,922,678 $ 70,813,926 
Total value of all transactions larger than $5,000 $ 29,024,824 $ 18,370,646 $ 170,808 
Percentage of total dollars for transactions larger 
than $5,000 17.52% 20.89% 0.24% 
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address new risks that might occur with acquisition cards. These 
additional risks might include the use of acquisition cards for 
high-dollar-value transactions and the use of some very high credit 
limits on these cards—one in particular with a limit of $1.5 million. 
This update would include establishing any limitations or controls over 
such transactions that are considered necessary to protect the interests 
of the government as a whole. 


1.46 Finally, while the randomly selected and high-dollar-value 
transactions of our audit did not identify any transactions that resulted 
in losses or abuse, we also examined the entire database of 2005 
transactions for the three departments. Using data mining and analysis 
software, we identified some questionable transactions, considering the 
names of the vendors and the nature of the transactions. We 
performed additional analysis and enquiry of a number of these 
transactions. In all cases, we found that the post-payment verification 
controls of the department, combined with monitoring practices of the 
service provider and the department, had resulted in the timely 
resolution of any situations that could have subjected the government 
to a loss of funds. 


1.47 We are aware that a risk of misuse exists in any program or 
payment method and that it is impossible for any control framework to 
be 100 percent accurate. However, the results of our examination of 
the controls for verifying and monitoring acquisition card transactions 
indicate that creating new or more controls is not the most essential 
step. Rather, it is essential to have departments apply their full 
attention and diligence to the task of ensuring that existing controls 
work as intended. 


1.48 Recommendation. Fisheries and Oceans Canada, Health 
Canada, and National Defence should ensure that, on a timely basis, 
all acquisition card transactions are properly verified and certified, 
according to section 34 of the Financial Administration Act and as 
stated in the Treasury Board Policy on Account Verification. 


Fisheries and Oceans Canada’s response. Fisheries and Oceans 
Canada agrees. Since the period covered by the scope of the audit, the 
Department has required that the cardholder’s supervisor or manager, 
who has the appropriate delegated signing authority, sign section 34 of 
the Acquisition Card Register. This practice will be monitored regionally. 


Health Canada’s response. All Health Canada acquisition card 
transactions are properly verified and certified in accordance with 
section 34; however, Health Canada will strengthen its acquisition 
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card policy to not permit cardholders to verify and certify, under 
section 34, transactions that the acquisition cardholders have 
processed. 


National Defence’s response. National Defence agrees. The 
Department has increased the level of post-payment compliance 
reviews to reinforce the importance of section 34 FAA verification 
and certification. This point is being emphasized as part of the formal 
training packages currently under development. 


Controls over monitoring travel card transactions and delinquencies vary 


1.49 Departments are responsible for authorizing, issuing, monitoring, 
and cancelling the use of travel cards and settling delinquent accounts. 
Employees are responsible for paying travel card balances and are 
reimbursed for business travel expenses, once they have completed a 
travel claim. 


1.50 One risk for the government could arise if the employee uses a 
travel card for non-business purposes and fails to pay the card account 
balance on time. This means that the government then becomes liable 
for the account balance. Another risk could arise if the government 
has reimbursed travel expenses to an employee, who then fails to pay 
the travel card account balance. The government can control both of 
these risks by monitoring travel cards closely to identify delinquent 
(overdue) accounts. We expected that departments would closely 
monitor the use of cards on a regular and timely basis to reduce these 
potential risks associated with travel cards, while taking into 
consideration the ability of the departments to manage the risk of 
default by using salary-offset provisions. 


151 We reviewed departments’ processes for monitoring transactions 
incurred on travel cards, and for following up on overdue accounts and 
recovering any amounts owing where necessary. We found that all 
three departments managed collection risk by monitoring accounts as 
they approached the point of default by the cardholder, including 
reviewing travel card transactions, notifying employees and their 
managers of the outstanding amounts, and requiring cardholders to 
pay any outstanding balances. However, on the travel card accounts 
that were not overdue, departments did not monitor or verify 
transactions on a regular basis to ensure that they represented only 
business expenses. 


152 The three departments also took corrective action at different 
times. We found that Fisheries and Oceans Canada monitored 
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delinquent accounts and took corrective action when accounts 
became 60 days overdue. The Department did the following: 


* it contacted delinquent cardholders at the 60-day mark, 


* it informed their respective managers of the status of accounts, 
and 


* it requested the employees to pay the balance. 


Health Canada and National Defence effectively monitored 
delinquent accounts only when they became 90 days overdue. We note 
that, according to the subcontract with the service provider, after 

90 days, the government has already become liable for the amount 
owing on the account. For Health Canada, we found that the number 
of delinquent accounts was minimal. This was not the case for 
National Defence, and therefore more active monitoring is required at 
that department to reduce the risk of liability to the government. This 
may also help reduce the amount of collection work that might be 
needed later on. 


1.53 Recommendation. National Defence, taking into account the 
risk of default, should more actively monitor its overdue travel card 
accounts. 


National Defence’s response. National Defence agrees. The 
Department is taking steps to increase the early review of delinquent 
designated travel cards centrally and through regional travel card 
coordinators. We will be proactive in determining the cause for 
delinquency and suspending cards at risk for default as they are 
identified. 


1.54 Departments periodically receive information on their card 
programs from service providers, central agencies, and internal sources 
such as internal audit or quality assurance reports. This information 
could range from the specifics of individual transactions to audits or 
reviews of the card programs. We expected that the departments would 
use this information, in a timely fashion, to assist in managing their 
acquisition and travel card programs and to ensure that they would 
achieve the intended benefits from these cards. 


Management information for monitoring acquisition card usage is exception-based 


1.55 As part of the contractual relationship between each department 
and its acquisition card service providers, the department has access to 
information or reports from the service providers, either electronically 


or in paper form, about its own spending on acquisition cards. We 
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found evidence that card coordinators reviewed these reports when 
they were provided in paper form. However, our interviews with staff 
indicated that the full range of information in the reports that are 
electronically available are not regularly accessed and reviewed. 


1.56 In all three departments we examined, we were informed that 
reporting to senior management was primarily done only as an 
exception and occurred only when 


° serious issues had arisen, 
* senior management requested information, or 


* internal audit or reviews were reported. 


1.57 In both Fisheries and Oceans Canada and Health Canada, we 
found little evidence of regular reporting to senior management on the 
acquisition card programs. However, we noted that Health Canada 
and National Defence had recently undertaken to work with the 
service providers to provide key information on how the acquisition 
card program is operating. In addition, at National Defence, we noted 
that the Department had created a “watch list” to monitor specific 
transactions—the results of which led to a 2004 internal audit on 
acquisition cards. 


1.58 Direct communication with the departmental card coordinator 
also appears to happen on the basis of exceptions. For example, when 
regional card coordinators identified issues, they did not always bring 
these issues to the attention of the departmental coordinator. Instead, 
they frequently reported the issues through their chain-of-command in 
the region. This practice can restrict the opportunity for program 
management to quickly become aware of significant issues and to 
ensure that relevant information is disseminated across the 
department. 


Management information for monitoring travel card usage is appropriate 


159 With respect to travel cards, American Express (AMEX) 
provides the departments with access to online activity reports. One 
report is issued per individual cardholder, which includes the number 
of days that a cardholder has had an account balance. Since the main 
risk for departments and the government with travel cards is the 
cardholder’s late payment of the account balances, this information in 
the activity report is sufficient to manage that risk. In addition, Public 
Works and Government Services Canada (PWGSC) oversees travel 
card usage, delinquent accounts, aging reports, and various other types 
of information on a government-wide basis. 
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Internal audit and reviews are performed 


160 Another source of information that management uses for 
monitoring purposes is internal audit reports. In all three departments 
we examined, internal audit had performed audits on the use of 
acquisition cards; these were useful to management in evaluating the 
performance of their acquisition card program. In 2004, internal audit 
at both Fisheries and Oceans Canada and Health Canada reported on 
acquisition cards. For Health Canada, a quality assurance review for 
the National Capital Region was reported in March 2006. 
Management action plans for these two departments have been 
developed to address concerns raised in these audits and reviews. At 
National Defence, the Chief of Review Services (CRS) performed 
some work on acquisition cards in 2004. However, at the time of our 
audit, the CRS internal audit report and action plan had still not been 
finalized. 


1.61 With respect to travel cards, the Audit and Evaluation 
Directorate at Fisheries and Oceans Canada conducted an audit 

in 2004 on the use of the travel card program. Neither of the other two 
departments has performed audits of travel cards, although both have 
performed audits of travel expenditures. 


Information on efficiencies achieved is not fully quantified 


1.62 Although there are no recent studies substantiating efficiencies 
achieved using acquisition cards, in 1998 the government issued a 
report that identified the acquisition card as the least costly payment 
method for processing purchases. The 1998 report does not provide a 
comprehensive analysis of the efficiencies achieved; however, the 
departments indicated to us that they feel they are in fact achieving 
efficiencies through the use of the cards. Some examples are as follows: 


* Each of the departments we examined had from 140,000 to 
391,000 transactions in 2005. Although this amounted to a total 
of about 750,000 acquisition card transactions, each department 
would have made only one payment per month to its respective 
service providers for the consolidated credit card statement 
received. Without the acquisition and travel cards, a payment 
would have to be made for almost every transaction, and thus 


there would be an increase in transactional costs. 


« A rebate incentive is provided to the government based on the 
dollar value of-acquisition and travel card expenditures incurred. 
In 2005, the rebates on the acquisition and travel cards provided 


to the government were in excess of $5 million. 
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1.63 Another compelling reason for using cards is the transactional 
information the acquisition and travel cards enable the government to 
gather and analyze. For example, this information helps the 
government and departments identify preferred vendors and can be of 
great use in negotiating with suppliers to achieve volume discounts and 
other efficiencies. 


1.64 There are also benefits to government employees in using travel 
cards while on travel status. These include both the convenience of 
obtaining cash advances to pay for travel costs and various insurance 
coverages. 


1.65 Although increased use of the acquisition cards is being 
encouraged in the departments we examined, they need to take into 
consideration that vendors may begin to incorporate the incremental 
cost of accepting payment by way of an acquisition card (merchant fee) 
into the price charged to the government. If this is the case, especially 
for high-dollar-value transactions, payment by way of an acquisition 
card could become more costly than using the traditional vendor 
invoice process. 


1.66 Asaresult, while there is anecdotal evidence of various 
efficiencies through the use of card programs, it is not possible to 
definitely conclude that the government has achieved its intended 
benefits from these programs. As the government expands the use of 
these programs for higher-dollar-value payments or for more payments 
in general, a more thorough analysis, comparing the benefits of these 
programs with the incremental costs of managing and monitoring 
them, is required. This analysis will make it possible to determine if the 
government is achieving the benefits in excess of the incremental costs 
of the programs. 


1.67 Recommendation. The Treasury Board Secretariat and Public 
Works and Government Services Canada should monitor developments 
regarding the evolving practices of acquisition and travel card programs 
to identify the benefits versus the incremental costs. 


The Treasury Board Secretariat’s and Public Works and 
Government Services Canada’s response. The Treasury Board 
Secretariat (TBS) and Public Works and Government Services 
Canada (PWGSC) agree with the recommendation. Consistent with 
our business practices, the TBS and PWGSC monitor current and 
evolving practices related to the acquisition and travel card programs 
through regular discussions with financial institutions, industry leaders, 
and other jurisdictions; attendance at conferences and workshops; and 
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review of current periodicals and articles. The TBS also plans to 
update the Report on Review of the Cost of Various Methods of Payment, 
which was originally issued in January 1998, to identify the costs 
related to the Government of Canada payment methodologies, 
including acquisition cards. The update of this report will provide 
departments with common information from which to develop their 
business cases and identify which payment options will result in the 
best benefits and savings to them. Forecasted completion of this 


update is 31 March 2009. 


Sharing of best practices Use of best practices by departments can be increased 


1.68 As part of our audit, we researched best practices in relation to 
the control and monitoring of acquisition card transactions and 
outstanding travel card balances, and we identified four best practices. 
Two of these best practices are applicable to both travel and acquisition 
cards (data mining, restriction of card use), while two (quality 
assurance and central billing) are mainly pertinent to acquisition cards. 


1.69 While the best practices for central billing and post-payment 
verification noted below are being used by many departments, data 
mining and restriction of card use are not frequently used. The use of 
these additional best practices can provide management with ongoing 
assurance that the programs are operating as expected. 


170 Data mining. This is an analytic process that involves searching 
large volumes of data for trends and patterns and unusual transactions. 
It is accomplished by evaluating a population of card transactions or 
account balances and bringing anomalies between different pieces of 
information to management’s attention. For example, parameters can 
be set to automatically identify trends or patterns, such as transactions 
done with questionable vendors, split transactions, or transactions of 
unusual amounts or relationships. Data mining is a tool that internal 
audit and management can easily implement to identify emerging 
problems of fraud, waste, or abuse. It can also provide statistics about 
purchases for management purposes, by product, and/or by vendor. 


1.71 Quality Assurance. This is a practice that allows management 
to re-examine monthly transactions and to monitor compliance with 
policies and procedures. The Treasury Board Policy on Account 
Verification establishes the requirement for departments to carry out 
post-payment verification and quality assurance activities. 


1.72 Restricting the use of cards. This offers numerous opportunities 
to control the use of cards. This tool allows management to limit, 
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based on a risk assessment, the vendors or the types of transactions 
that can be processed on a given card. This provides better control 
over the card and possibly fewer problematic transactions. 


1.73 Central billing practice. This allows a department to issue one 
payment for an entire volume of purchases to the respective card 
service providers. Departments are now able to download all 
transactional data from the service providers directly into their 
departmental financial systems on a regular (often daily) basis. This 
allows departmental staff to review or validate the transactions on a 
more immediate basis. This process, when combined with a rigorous 
post-payment verification process, allows departments to make one 
payment to the service providers in a controlled and immediate 
fashion, thus maximizing rebates. The Treasury Board Secretariat 
highlighted the practice of central billing as one of the best practices 
followed by all three departments. However, we understand that 
centralized billing has not yet been implemented throughout the 
government. We also understand that the centralized billing practice 
could be used by departments for their designated-responsibility-centre 
travel card accounts. This could help to ensure timely payments, 
maximize rebates, and avoid interest charges. 


1.74 Departmental coordinators also indicated to us that they feel it 
would be useful to share best practices with their colleagues from other 
departments. Methods of disseminating information among managers 
and users in departments could include, for example, symposia and a 
shared Internet site containing information, emails, or newsletters. 
The Interdepartmental Travel Users Working Group has developed a 
system for sharing best practices in the use of travel cards. It is 
important that a similar method be developed for sharing best 
management practices for acquisition cards to increase the use of the 
advantages offered by these programs, improve the application of 
control procedures throughout the government, and minimize each 
department’s risk of incurring problematic transactions. 


175 Recommendation. Departments should make better use of best 
practices, such as data mining, quality assurance, restricted use, and 
central billing in managing and monitoring their acquisition card 
transactions and outstanding travel card balances. 


Fisheries and Oceans Canada’s response. Fisheries and Oceans 
Canada agrees. The Department has already developed and 
implemented best practices related to central billing and quality 
assurance. The Department’s outstanding travel card balances are 


closely monitored, and delinquent accounts are minimal. 
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The Department is open to participating in any forums, including 
interdepartmental sessions to improve on its practices. In 2007-08, the 
Department will gather additional information on best practices 
related to data mining and restricted use, and implement these 
practices, where practical. 


Health Canada’s response. Health Canada will continue to make 
use of best practices, where possible. 


National Defence’s response. National Defence agrees. The 
Department will continue to expand the use of best practices by 
building on the monthly reports already in place for the Senior Full- 
Time Financial Officer. This will include enhancing the level of real- 
time transaction information available to cardholders through 
increased access to the online portals for data mining. In 
conjunction with the implementation of revised standing offers for 
common goods and services, regional coordinators will also be 
provided with more information on their authority to restrict card use, 
where appropriate, to particular commodities and potentially to 
particular vendors. 


1.76 Recommendation. The Treasury Board Secretariat and Public 
Works and Government Services Canada should establish a forum for 
sharing best practices for acquisition cards across government. 


The Treasury Board Secretariat’s and Public Works and 
Government Services Canada’s response. The Treasury Board 
Secretariat (TBS) and Public Works and Government Services Canada 
(PWGSC) agree with the recommendation. The TBS and PWGSC 
recognize the importance of sharing best practices and will examine 
ways to share and communicate leading practices for acquisition cards 
across the government. For example, PWGSC held its banking 
colloquium on 5 March 2007, which included a workshop on best 
practices related to acquisition card programs. An update of PWGSC's 
Receiver General Manual, set for 31 March 2008, will contain a chapter 


on acquisition cards that will include leading practices. 


Conclusion 


1.77 The travel card program in the three departments that we 
examined has a control framework in place, which addresses the risks 
associated with that program. The acquisition card program in the 
three departments we examined has a control framework in place that 
relies significantly on post-payment verification and certification to 
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ensure the proper authorization of card transactions. The results of our 
audit of a sample of acquisition card transactions did not identify any 
losses or abuse. However, the control frameworks over acquisition 
cards were not applied in a rigorous manner. It is important that 
departments strengthen their control over the use of acquisition cards 
by dealing with the weaknesses—particularly the inconsistent 
application of verification and certification (as specified in section 34 
of the Financial Administration Act), which we noted in this audit. 
Fixing them will become more crucial, given that departments are 
interested in using the acquisition cards to a greater extent. As the use 
of cards increases, so do the risks that must be managed. 


178 There is anecdotal evidence that efficiencies have been 
achieved. Nevertheless, in the absence of a comprehensive analysis of 
incremental costs and benefits, it is not possible to conclude that the 
government knows whether it has achieved the intended benefits from 
these programs. 
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About the Audit 


Objective 


The objective of this audit was to determine whether the government has an appropriate management 
control framework over the use of acquisition and designated travel cards and if the government knows 
whether it is achieving the intended benefits from these programs. 


Scope and approach 


In our audit, we compared departmental policies, procedures, and practices to determine if they were clear, 
consistent with the Treasury Board policies, and communicated properly to the parties who are involved in 
managing and using the cards, so that they are fully aware of their roles and responsibilities. 


We assessed, through the use of review and enquiry, whether departments have received, processed, and 
disseminated information about the use of the cards in a timely manner. 


We examined the database of transactions for each of the three departments and tested transactions to 
determine 


+ whether departmental systems of control were in place, which would ensure probity and prudence in 
the use of cards; 


- how departments were applying key sections of the Financial Administration Act; and 
- how departments were undertaking ongoing monitoring. 
We identified the following high-value transactions: 


23 transactions over $100,000 for Fisheries and Oceans Canada, 


1 transaction over $8,000 for Health Canada, and 


7 transactions over $100,000 for National Defence. 


Of these high-value transactions, we examined a total of 21: 
« 13 transactions from Fisheries and Oceans Canada, 
- | transaction from Health Canada, and 
« 7 transactions from National Defence. 


From the remainder of the population of transactions, we randomly selected non-proportional stratified 
samples. For all three departments, we selected samples in such a way that large-dollar transactions would 
be proportionally over-represented. All proportions that we are reporting are therefore appropriately 
weighted to reflect an accurate estimate of findings for the entire population. We selected samples sizes in 
such a way as to give confidence intervals that would be less than 10 percent at a 90 percent level of 
confidence. This means that, for this chapter, the margin of sampling error is plus or minus 10 percent, 


valid 18 times out of 20. 


Our audit covered transactions during the period from | January 2005 to 31 December 2005. 


Chapter 1 | 25 


Report of the Auditor General of Canada—May 2007 


USE OF ACQUISITION AND TRAVEL CARDS 


The departments included in the audit were 
- Fisheries and Oceans Canada, 
- Health Canada, and 


« National Defence. 


We also examined the role played by the Treasury Board Secretariat and Public Works and Government 
Services Canada in these programs. 


Criteria 


We expected that the Treasury Board Secretariat and Public Works and Government Services Canada 
would have 


* established directives and guidance and best practices on credit cards and expenditure management; 


exercised oversight of the use and management of those cards; and 


ensured that the government was receiving the financial benefits and reporting requirements from the 
card companies, according to the applicable agreements. 


We expected that departments would have 


e 


established policies and procedures that were clear and consistent with the Treasury Board policies; 


developed communication and training practices to ensure that parties involved in managing and 
using the cards are fully aware of their roles and responsibilities; 


implemented systems of control to ensure probity and prudence in using their cards and that the 
system of control was functioning as intended; 


received, processed, and disseminated information related to the use of cards in a timely manner to 
allow for proper management of the cards; 


established regular monitoring practices for the cards, including reviews and audits that would be 
performed in a timely and complete manner; and 


taken advantage of all financial benefits associated with the cards. 


Audit work completed 


The audit work for this chapter was substantially completed by 29 September 2006. 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 1. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


1.19 The Treasury Board Secretariat 
should update the policy with respect to 
acquisition cards including whether 


“corporate” acquisition cards should be 
permitted. (1.15—1.18) 


1.24 The Treasury Board Secretariat 


should complete and issue its directive 
on travel cards. (1.20-1.23) 


1.25 Public Works and Government 
Services Canada should provide 
departments and agencies with 
operational guidelines pertaining to the 
provisions of the contract with the 
travel card service provider, including 
guidelines pertaining to the 
responsibility of the government for 
unpaid travel card balances. 


(1:20-1.23) 


The Treasury Board Secretariat’s response. The Treasury 
Board Secretariat agrees with the recommendation. As part of 
the renewal of Treasury Board’s financial management policies, 
the Directive on Acquisition Cards is being updated. The revised 
directive will not permit the issuance of acquisition cards in the 


name of an organization or asset, such as a vessel, as it would 
have an impact on the government’s ability to monitor and 
control the usage of the card as well as on the merchant’s ability 
to accept the card as payment. Forecasted completion for the 


updated directive is 31 March 2008. 


The Treasury Board Secretariat’s response. The Treasury 
Board Secretariat agrees with the recommendation. As part of 
the renewal of Treasury Board’s financial management policies, 
the Directive on Travel Cards and Travellers Cheques is being 
updated. Forecasted completion for the updated directive is 


31 March 2008. 


Public Works and Government Services Canada’s response. 
Public Works and Government Services Canada agrees with this 
recommendation and will provide departments and agencies 
with operational guidelines pertaining to the provisions of the 
contract with the travel card service provider. 
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1.31 Fisheries and Oceans Canada, 
Health Canada, and National Defence 
should formalize training for regional 
and departmental coordinators and 
quality assurance personnel to ensure 
that information about all issues related 
to acquisition and travel card 
management within a department is 
understood by those personnel, and 
that the information is disseminated to 
program managers. (1.26—1.30) 


Verification and monitoring of card transactions 


1.48 Fisheries and Oceans Canada, 
Health Canada, and National Defence 
should ensure that, on a timely basis, all 
acquisition card transactions are 
properly verified and certified, 
according to section 34 of the Financial 
Administration Act and as stated in 

the Treasury Board Policy on Account 
Verification. (1.32-1.47) 
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Fisheries and Oceans Canada’s response. Fisheries and 
Oceans Canada agrees. Over the next 12 months, the 
Department will implement a training program for card 
coordinators and quality assurance personnel, and will improve 
the departmental coordination of card management and of the 
quality assurance function. In addition, the Department will 
take the necessary steps to inform card coordinators of the 
availability on the website of training material and tools 
accessible to them. 


Health Canada’s response. Health Canada agrees that training 
should be available for these groups and will explore 
alternatives for the delivery of training in respect of 
departmental coordinators, including the possible provision of 
this training through central agencies or the Canada School of 
Public Service. 


National Defence’s response. National Defence agrees. The 
Department is developing a range of training packages for all 
personnel involved in the use of acquisition and travel cards. It is 
expected that the formal training program will be implemented 
through regional coordinators over the course of 2007. 


Fisheries and Oceans Canada’s response. Fisheries and Oceans 
Canada agrees. Since the period covered by the scope of the 
audit, the Department has required that the cardholder’s 
supervisor or manager, who has the appropriate delegated 
signing authority, sign section 34 of the Acquisition Card 
Register. This practice will be monitored regionally. 


Health Canada’s response. All Health Canada acquisition 
card transactions are properly verified and certified in 
accordance with section 34; however, Health Canada will 
strengthen its acquisition card policy to not permit 
cardholders to verify and certify, under section 34, 
transactions that the acquisition cardholders have processed. 


National Defence’s response. National Defence agrees. The 
Department has increased the level of post-payment compliance 
reviews to reinforce the importance of section 34 FAA 
verification and certification. This point is being emphasized as 
part of the formal training packages currently under 
development. 
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Recommendation = 


1.53 National Defence, taking into 
account the risk of default, should more 
actively monitor its overdue travel card 
accounts. (1.49-1.52) 


National Defence’s response. National Defence agrees. The 
Department is taking steps to increase the early review of 
delinquent designated travel cards centrally and through 
regional travel card coordinators. We will be proactive in 
determining the cause for delinquency and suspending cards at 
risk for default as they are identified. 


Information for managing card programs 


1.67. The Treasury Board Secretariat 
and Public Works and Government 
Services Canada should monitor 
developments regarding the evolving 
practices of acquisition and travel card 
programs to identify the benefits versus 
the incremental costs. (1.54—1.66) 


The Treasury Board Secretariat’s and Public Works and 
Government Services Canada’s response. The Treasury Board 
Secretariat (TBS) and Public Works and Government Services 
Canada (PWGSC) agree with the recommendation. Consistent 
with our business practices, the TBS and PWGSC monitor 
current and evolving practices related to the acquisition and 
travel card programs through regular discussions with financial 
institutions, industry leaders, and other jurisdictions; attendance 
at conferences and workshops; and review of current periodicals 
and articles. The TBS also plans to update the Report on Review 
of the Cost of Various Methods of Payment, which was originally 
issued in January 1998, to identify the costs related to the 
Government of Canada payment methodologies, including 
acquisition cards. The update of this report will provide 
departments with common information from which to develop 
their business cases and identify which payment options will 
result in the best benefits and savings to them. Forecasted 
completion of this update is 31 March 2009. 


Sharing of best practices 


175 Departments should make better 
use of best practices, such as data 
mining, quality assurance, restricted 
use, and central billing in managing and 
monitoring their acquisition card 
transactions and outstanding travel 


card balances. (1.68-1.74) 


Fisheries and Oceans Canada’s response. Fisheries and Oceans 
Canada agrees. The Department has already developed and 
implemented best practices related to central billing and quality 
assurance. The Department’s outstanding travel card balances 
are closely monitored, and delinquent accounts are minimal. 


The Department is open to participating in any forums, 
including interdepartmental sessions to improve on its practices. 
In 2007-08, the Department will gather additional information 
on best practices related to data mining and restricted use, and 
implement these practices, where practical. 


Health Canada’s response. Health Canada will continue to 
make use of best practices, where possible. 
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1.76 The Treasury Board Secretariat 
and Public Works and Government 
Services Canada should establish a 
forum for sharing best practices for 
acquisition cards across government. 


(1.68-1.74) 
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National Defence’s response. National Defence agrees. The 
Department will continue to expand the use of best practices by 
building on the monthly reports already in place for the Senior 
Full-Time Financial Officer. This will include enhancing the 
level of real-time transaction information available to 
cardholders through increased access to the online portals for 
data mining. In conjunction with the implementation of revised 
standing offers for common goods and services, regional 
coordinators will also be provided with more information on 
their authority to restrict card use, where appropriate, to 
particular commodities and potentially to particular vendors. 


The Treasury Board Secretariat’s and Public Works and 
Government Services Canada’s response. The Treasury Board 
Secretariat (TBS) and Public Works and Government Services 
Canada (PWGSC) agree with the recommendation. The TBS 
and PWGSC recognize the importance of sharing best practices 
and will examine ways to share and communicate leading 
practices for acquisition cards across the government. For 
example, PWGSC held its banking colloquium on 

5 March 2007, which included a workshop on best practices 
related to acquisition card programs. An update of PWGSC's 
Receiver General Manual, set for 31 March 2008, will contain a 
chapter on acquisition cards that will include leading practices. 
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Development Canada and Canada 
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All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Main Points 


The Canada Student Loans Program of Human Resources and Social 
Development Canada (HRSDC) provides loans, Canada Access 
Grants, and Canada Study Grants to qualified students who need 
financial assistance for post-secondary education. The grants and the 
Millennium Bursary and Access Bursary programs of the Canada 
Millennium Scholarship Foundation are designed to help limit the 
amount of debt that recipients accumulate. 


We examined whether the Department and the Foundation have 
appropriate controls to ensure that they manage their responsibilities 
with due care and in compliance with legislation and other 
requirements. We looked at the controls that the Canada Revenue 
Agency has in place to ensure that the collection of defaulted Canada 
Student Loans complies with requirements. We also examined whether 
the Department and the Foundation ensure that information on the 
programs is easy to find and understand and that services are accessible 
and convenient. We examined how they measure their programs’ 
performance and report results. We did not examine the activities of 
the other partners in delivering the loan, grant, and bursary 
programs—the provinces and territories, and two private sector service 
providers contracted by HRSDC to handle most day-to-day operations 
of the Canada Student Loans Program. 


The federal government has said that Canada’s international 
competitiveness depends on having an educated and skilled labour 
force. To improve access to higher education is the main objective of 
both the Canada Student Loans Program and the Foundation. To this 
end, HRSDC loaned $1.9 billion to some 350,000 post-secondary 
students in 2005—06 and awarded 78,500 grants amounting to about 
$129 million. In 2006 the Foundation awarded about $298 million in 
Millennium Bursaries to some 100,000 students and over $40 million 
in Access Bursaries to more than 20,000 students. 


- With a few exceptions, the Canada Student Loans Program is 


well managed. The Department has the necessary controls in place 


for disbursing loans and grants; it also obtains reasonable assurance 


Chapter 2 1 


FEDERAL LOANS AND GRANTS FOR POST-SECONDARY EDUCATION— 
HUMAN RESOURCES AND SOCIAL DEVELOPMENT CANADA AND CANADA MILLENNIUM SCHOLARSHIP FOUNDATION 


that its partners in delivering the Program are fulfilling their 
responsibilities. However, it does not have a clear strategy for 
measuring whether the Program has improved access to 
post-secondary education. Further, the Department has postponed 
completing its evaluation of the Program from 2006 to 2011. 


The Foundation’s bursary programs are well managed. The 
Foundation has the necessary controls in place to ensure that 
eligibility for bursaries is assessed correctly and that payments are 
issued in the right amounts to the right people. It also obtains 
reasonable assurance that the provinces and territories are fulfilling 
their responsibilities in delivering the programs. The Foundation 
collects adequate information to report its achievements against 
objectives, and it meets its obligation to report the results of its 
activities to Parliament. 


The Department and the Foundation are taking steps to make 
prospective post-secondary students and their families more aware of 
the financial assistance available to them. The Department has also 
improved its communication to students about measures available to 
help manage their debts. 


The Canada Revenue Agency monitors the performance of private 
collection agencies, and Human Resources and Social Development 
Canada monitors the consistency of the information the agencies 
provide. However, neither organization adequately monitors the 
agencies’ compliance with key collection requirements of the 
Canada Student Loans Program. The Canada Revenue Agency has 
contracted with private collection agencies to handle about 

$450 million of the $800 million in defaulted loans owed to HRSDC 
(at 31 March 2006); it handles the remaining $350 million itself. 


HRSDC and the Foundation have developed a good research 
relationship with one another. However, they do not have adequate 
arrangements for consulting each other to achieve their shared 
objective of improved access to post-secondary education. 


Human Resources and Social Development Canada, the Canada 
Revenue Agency, and the Canada Millennium Scholarship 
Foundation have responded. The Department, Agency, and 
Foundation have agreed with each of our recommendations and have 
committed to taking action. Their responses are included throughout 
the chapter. 
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Introduction 


2.1 In its budgets of 2004, 2005, and 2006 the federal government 
observed that a more educated and skilled labour force is essential to 
Canada’s current and future international competitiveness. The 
government sees investments in education and training as critical to 
productivity and economic growth. Although education falls within 
the legislative jurisdiction of the provinces and territories, the federal 
government manages or contributes to several programs that provide 
financial support for post-secondary education. 


2.2. Our audit looked at the Canada Student Loans Program (CSLP) 
of Human Resources and Social Development Canada (HRSDC) and 
at the Millennium Bursary and Access Bursary programs of the Canada 
Millennium Scholarship Foundation. These programs provide direct 
financial support to students. The CSLP and the Foundation’s 
programs share the same objective: to improve access to 
post-secondary education for students in financial need. Other federal 
programs supporting post-secondary education that are not part of this 
audit include education savings programs, research funding, tax 
incentive measures, programs to support Aboriginal post-secondary 
education, and the Canada Social Transfer. 


The Canada Student Loans Program 


2.3 The Government of Canada established the Canada Student 
Loans Program in 1964 as a statutory spending program under the 
Canada Student Loans Act. In 1994, the Canada Student Financial 
Assistance Act came into force: it governs the provision of loans and 
other forms of financial assistance to students. The Program provides 
loans and/or grants to qualified students attending post-secondary 
institutions. The loans and grants are based on financial need, 
regardless of province or territory of residence. Since 2000, the federal 
government has delivered loans directly; previously it did so through 
financial institutions. In 2005—06, HRSDC loaned $1.9 billion to some 
350,000 post-secondary students. For direct loans, there is currently an 
outstanding debt of about $8.2 billion borrowed by 990,000 students. 


2.4 Quebec, the Northwest Territories, and Nunavut have chosen 
not to participate in the Program. Instead, under the Canada Student 
Financial Assistance Act, they receive transfer payments from the 
federal government (a total of about $160 million for 2005-06). 
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25 Human Resources and Social Development Canada works 
directly with several partners (Exhibit 2.1) to deliver the Canada 
Student Loans Program: the participating provinces and territory, 

the Canada Revenue Agency, Public Works and Government Services 
Canada (PWGSC), and two private-sector service providers it has 
contracted to administer most of the day-to-day operations for the 
Program. The Learning Branch of HRSDC is responsible for developing 
and implementing the policies, regulations, procedures, and criteria for 
determining student eligibility for loans under the Program. 


26 Under the Canada Student Financial Assistance Act, the Minister of 
Human Resources and Social Development has designated the Ministry 
of Education (or higher education) in each participating province or 
territory as the appropriate authority to administer parts of the Act and 
the Canada Student Financial Assistance Regulations. The Minister has 
also entered into an administrative fees arrangement with each 
participating province and territory for the eligibility assessment services 
it provides. In 2005-06, the total amount in administrative fees paid by 
the federal government was about $14 million. 


27 Students apply for both a federal loan and a provincial or 
territorial loan at the same time. Eligibility for a federal loan is assessed 
in conjunction with the provincial and territorial loan application. The 
federal part of a student loan usually makes up about 60 percent of the 
loan total. The provincial or territorial authorities serve as a student’s 
single point of contact in applying for a loan and having it assessed. 


28 HRSDC contracts two private-sector service providers to 
perform most of the day-to-day operations for the Canada Student 
Loans Program. One service provider serves students attending public 
educational institutions and the other serves students in private 
institutions. They process the loan documents received from the 
participating provinces and territory (Yukon) and send transactions 

for final authorization to HRSDC. The Department then forwards 
approved transactions to PWGSC to issue and disburse the federal loan 
payments. The service providers are responsible for managing accounts 
from disbursement of the loan to repayment and for maintaining 
account information. They also administer the National Student Loans 
Service Centre, which deals with most student telephone enquiries or 
complaints. In 2005-06, HRSDC paid $50 million in administration 
fees to the service providers. The Department’s current contracts with 
the service providers expire at the end of March 2008. During our 
audit, the Department was reviewing responses to its request for 
proposals to find a single service provider. A contract has since been 
awarded, and the transition will occur in mid-March 2008. 
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Bad debt expense—An allowance for the cost 
to the Canada Student Loans Program of future 
defaulted loans and debt relief measures. 


29 Students are to begin repaying their loans no later than 

six months after their studies end. For full-time students, no interest 
accrues during their studies; for part-time students, interest accrues 
and is payable during their studies. For some provinces and the Yukon, 
the service providers process repayments of only the federal student 
loans. A student must repay the province or territory separately for its 
portion of the loan. However, HRSDC has signed integration 
agreements with four provinces, accounting for 60 percent of CSLP 
borrowers. Under these agreements, the service providers are a single 
point of contact for repaying both the federal and provincial portions 
of a loan. These integration agreements also create common 
disbursement processes and debt management programs. 


210 Under the Canada Student Loans Program, HRSDC also awards 
Canada Access Grants and Canada Study Grants. A student may be 
eligible to receive a combination of Access and Study Grants. A 
student receiving an Access or Study Grant receives less money in the 
form of a repayable loan, thus reducing debt. In most cases, provinces 
and the territory assess eligibility for Access and Study Grants when a 
student applies for a Canada Student Loan. In 2005-06, HRSDC 
awarded $55 million in Access Grants to over 29,000 students and 
$74 million in Study Grants to 49,500 students. 


211 The Canada Student Loans Program also provides assistance to 
students facing repayment difficulties: Students may have the terms of 
their loan revised; interest relief is available for up to 54 months, 
during which time no interest accrues to the student and no payments 
are required; and debt reduction in repayment is available to certain 
borrowers who experience exceptional difficulties in meeting their loan 
obligations. If a student’s loan is in arrears for 270 days, the Canada 
Revenue Agency’s Non-Tax Collection Services will start collection. 


212 In its 2005-06 Departmental Performance Report, HRSDC 
states that its costs and fees for the Canada Student Loans Program 
(direct loans) were $993 million for the year. This includes a bad debt 
expense of just over $310 million. The interest revenue was 


$316 million. 
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Conditional grant—A transfer payment that is 
approved by Parliament and made available to 
an organization as a lump sum, paid up front, to 
cover expenses over a number of years. The 
Treasury Board requires the sponsoring 
department to report on the organization's 
results within its annual departmental 
performance report and situate these results 
within the overall plans and priorities of the 
department. 
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The Millennium Bursary and Access Bursary programs 


2.13 The Budget Implementation Act, 1998, established the Canada 
Millennium Scholarship Foundation to provide assistance to students 
in financial need. The Minister of Human Resources and Social 
Development tables the Foundation’s annual report (including audited 
financial statements) in Parliament. However, unlike the Department, 
the Foundation is not answerable to Parliament through the Minister. 


2.14 The federal government provided the Canada Millennium 
Scholarship Foundation with a conditional grant of $2.5 billion 

in 1998 under a funding agreement signed by the Minister of Human 
Resources Development (the title at that time) and the Minister of 
Finance. This agreement is a legally binding contract that places 
obligations on both departments and the Foundation. The 
Foundation’s enabling legislation also structures how the Foundation is 
to administer its funds and award its bursaries. 


2.15 Under the Millennium Bursary and Access Bursary programs, the 
Foundation provides bursaries to undergraduate students based on 
financial need. In most provinces, a bursary is applied to the loan to 
reduce the debt; in some provinces and the territories, it is added, or 
part of it is added, to the amount of the student loan. 


2.16 Students may be considered for a Millennium Bursary when they 
apply for student financial assistance from their province or territory of 
residence. In some jurisdictions, they must indicate on the student 
assistance application that they wish to be considered for the bursary; 
in others, they are considered automatically. Every year, beginning in 
1999-2000 and ending in 2008-09, the Foundation expects to provide 
about 90,000 eligible students with about $3,000 each. At 

31 December 2006, the cumulative value of Millennium Bursaries was 
almost $2 billion, including just over $298 million paid in 2006 to about 
100,000 students across Canada. 


2.17. In 2005, the Foundation introduced Millennium Access 
Bursaries to reach specifically targeted populations. Beginning 

in 2005-06 and ending in 2008-09, the Foundation expects to provide 
about 25,000 eligible students with up to a total of $50 million a year 
in Access Bursaries. In 2006, it awarded over $40 million in Access 


Bursaries to more than 20,000 students. 


2.18 Provinces and territories carry out the eligibility assessment 
for the Foundation’s bursaries, as they do for the federal portion of 
student loans. The Foundation has a separate agreement with each 


of the 10 provinces and the 3 territories to process its bursaries. 


T 


Report of the Auditor General of Canada—May 2007 


Chapter 2 


FEDERAL LOANS AND GRANTS FOR POST-SECONDARY EDUCATION— 
HUMAN RESOURCES AND SOCIAL DEVELOPMENT CANADA AND CANADA MILLENNIUM SCHOLARSHIP FOUNDATION 


Control framework—Canada 
Student Loans Program 


It pays administration fees for the eligibility assessment services they 
provide; in 2006, the fees were about $2.8 million. 


Focus of the audit 


2.19 Our objective was to determine whether federal loan and debt 
reduction programs for post-secondary students are managed to 
improve access to post-secondary education while ensuring that the 
necessary controls are in place. 


220 The audit focused on how HRSDC and the Canada Millennium 
Scholarship Foundation ensure that 


* they have adequate controls to manage the loan, bursary, and 
grant programs with due care and in compliance with legislation 
and other requirements; 


¢ their efforts to increase awareness and use of the programs are 
effective; 


¢ their services to clients are accessible and convenient; 


¢ they have satisfactory procedures to collect performance 
information, and decision makers use this information to achieve 
program objectives; and 


* they provide meaningful reports to Parliament as required. 


2.21 We also examined how the Canada Revenue Agency ensures 
that controls are in place for collecting defaulted loans according to 
agreements and other requirements. 


2.22 More details on the audit objective, scope, approach, and criteria 
are in About the Audit at the end of this chapter. 


Observations and Recommendations 


2.23 Each organization involved in the Canada Student Loans 
Program is responsible for implementing its own controls to ensure 
compliance with federal requirements in processing loans and grants. 
Human Resources and Social Development Canada (HRSDC) has an 
oversight role to ensure that all third parties, that is, organizations 
outside the federal government, are exercising due care in fulfilling 
their responsibilities. For collection of defaulted loans, the Canada 
Revenue Agency also has an oversight role. 
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2.24 Therefore, for each phase of the loan and grant life cycle, we 
expected HRSDC and the Canada Revenue Agency to have 
implemented control structures that include the following elements: 


* risk assessments, so that controls are designed and adapted based 
on key risks; 


¢ clear roles and responsibilities; 


* clear communication of federal requirements, including policies 
and procedures; and 


* ameans to provide HRSDC and the Canada Revenue Agency 
with reasonable assurance that third parties are fulfilling their 
responsibilities. 


2.25 The delivery structure and controls for the Canada Study Grants 
are different from those for the Canada Student Loans and Access 
Grants. Thus, we report separately on the Study Grants later in the 
chapter. 


Program risks are assessed 


2.26 Management of the Canada Student Loans Program conducts 
annual risk assessments of the Program. In 2006, the assessments were 
extended to include the Canada Access Grants. A joint federal/ 
provincial/territorial risk assessment of measures used for determining 
loan eligibility was conducted in 2004. As a result of these risk 
assessments, the Department has taken initiatives aimed at 
strengthening controls for processing the loans and grants, such as 
developing additional policies and procedures and expanding its audits 
of service providers. 


Roles, responsibilities, and federal requirements are clearly defined and 
communicated 


2.21 We found that HRSDC has clearly defined the roles and 
responsibilities of the organizations involved in delivering the Canada 
Student Loans and Access Grants. It has also developed appropriate 
policies and procedures to support the administration of the loans and 
grants, from assessing eligibility to reimbursing loans. These are 


updated periodically as risks emerge and issues are identified. 
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Designation—tThe process by which 
post-secondary education institutions obtain the 
recognition from the provincial and territorial 
governments that allows their students to be 
eligible for student loans. 


2.28 HRSDC has delegated responsibility to the Ministry of 
Education (or higher education) in the nine participating provinces 
and the Yukon for 


¢ designating educational institutions, 
¢ determining students’ eligibility and financial needs, and 


* issuing loan and grant certificates. 


2.29 The ministries’ roles and responsibilities in these areas are 
defined in the Canada Student Financial Assistance Act and Regulations. 
HRSDC has provided the participating provinces and territory with 
policies and procedures that further define the federal requirements set 
out in the Act and Regulations for assessing students’ eligibility and 
financial needs. 


2.30 The Department’s contracts with the service providers clearly 
establish their roles and responsibilities. They set out in detail the 
procedures for managing the loans and grants, such as initiating the 
loan and grant disbursements, processing loan reimbursements, and 
maintaining the master student files. The Department has also 
developed additional guidelines on federal requirements in specific 
areas, such as use of debt reduction measures. 


HRSDC obtains reasonable assurance that third parties are fulfilling their 
responsibilities 


2.31 Audits and reviews. HRSDC conducts periodic audits and 
reviews of how the service providers are managing the loan and grant 
process. We found that it has conducted these audits and reviews 
appropriately and used their results to improve the controls for 
processing the loans and grants. 


2.32. Over the years, HRSDC has adapted the audit and review 
process to meet new requirements and address emerging risks. For 
example, it expanded the scope of the audits and reviews to include 
the Canada Access Grants, and it added certain audit criteria and 
removed others to better focus on risks. 


2.33 HRSDC also examines reports by provincial auditors general on 
student financial assistance programs, as independent sources of 
information on whether the provinces use adequate processes to assess 
students’ eligibility and needs. We found that management of the 
Program reviews the observations arising from these reports to 
determine whether the Department needs to take action. 
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Reconciliation—A comparison between two 
sets of data to determine if they agree and to 
account for any differences. 
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2.34 The provinces, territory, and service providers handle most of 
the delivery of the loans and grants. HRSDC is responsible for 
disbursing the funds and maintaining the database on loans in default. 
To ensure that the loan and grant information it processes is accurate, 
HRSDC periodically reconciles its own financial information with the 
information of the service providers and PWGSC. We found that it 
does this properly. 


2.35 Joint committees and working groups. The Canada Student 
Financial Assistance Act gives the Minister of Human Resources and 
Social Development the authority to enter into agreements with 
provincial and territorial governments and to give them binding 
directives. However, there are no formal agreements for administering 
the eligibility and needs assessments for Canada Student Loans and 
Access Grants (apart from certain clauses in one of the integration 
agreements). Instead, HRSDC uses joint committees and working 
groups with the provinces and territory to share best practices and 
improve the consistency and adequacy of processes and controls for 
assessments of students’ eligibility and financial needs. This 
collaboration has led to such key initiatives as 


¢ establishing minimum criteria to designate educational 
institutions as eligible for government financial assistance 
programs for post-secondary students; 


* completing joint audits with several provinces to assess, among 
other things, how prepared an educational institution is to comply 
with the designation requirements, and planning for systematic 
audits of educational institutions starting in 2008; and 


* developing a draft framework to standardize measures for verifying 
loan eligibility—for example, validating students’ identity and 
income with third parties. 


Information on controls for eligibility and needs assessments is not well 
documented 


2.36 Through the federal/provincial/territorial committees and 
working groups, as well as its review of the provincial auditor generals’ 
reports, HRSDC is informed of risks, challenges, and controls related 
to eligibility and needs assessments carried out by the provinces and 
territory. Program officials told us that they consider these risks, 
challenges, and controls in determining what action, if any, is needed. 
However, we found little documentation on whether they concluded 
that the controls were adequate or on what, if any, related action was 


taken. For example, the Department developed an inventory of 
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Integrity—Assurance that the right Canada 
Student Loans Program client receives the right 
benefit or service at the right time and for the 
intended purpose, while client information is 
safeguarded and public confidence is 


maintained. 


provincial integrity measures in 2004, but there was no record of 
whether it considered the measures to be satisfactory. Nor has the 
inventory been updated even though some of the measures have since 
been changed. 


2.37 Asa good management practice and to minimize the risk that 
the Department may lose important knowledge when employees leave, 
a formal process is needed to manage this information. 


2.38 Recommendation. To ensure that the knowledge acquired 
about the provinces’ and the territory’s processes for eligibility and 
needs assessments is available for managing the Canada Student Loans 
Program, Human Resources and Social Development Canada should 
document its conclusions on the adequacy of controls and any related 
action taken. 


Human Resources and Social Development Canada’s response. The 
Department agrees that the Canada Student Loans Program would 
benefit from better documentation of provincial and territorial 
processes for eligibility and needs assessment. Program officials already 
meet regularly with officials from participating jurisdictions to share 
information on their respective controls of these processes. The 
Department will institute procedures to ensure that its conclusions on 
the adequacy of the Program’s controls for eligibility and needs 
assessments are documented. 


A formal approach to monitoring collection procedures is needed 


2.39 In August 2005, the responsibility for collecting debts owed to 
various HRSDC programs, including student loans in default, was 
transferred from the Department to the Canada Revenue Agency 
(CRA), together with the personnel assigned to these activities. The 
Department and the Agency signed agreements specifying the roles 
and responsibilities of each party. HRSDC continues to maintain the 
database on student loans in default, and it remains responsible for 
recording the amounts recovered and applying them to the correct 
borrower account. With the transfer, the CRA has continued to move 
forward with an extensive modernization strategy, begun by HRSDC, 
to improve the recovery of debts owed. 


2.40 The CRA’s Non-Tax Collection Services, which operates 
separately from its Tax Collection Services, carries out the collection 
activities in two regional offices (Belleville, Ontario and Vancouver, 
British Columbia). The CRA also has contracts with 12 private 
collection agencies. Of the $800 million in direct student loans in 
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default at 31 March 2006, about $450 million was handled by the 
private collection agencies. The CRA’s regional offices handled the 
remaining $350 million. 


2.41 Monitoring compliance of CRA staff with collection 
procedures. As part of our audit, we visited the Belleville regional 
office, which handles about 60 percent of all the student loan accounts 
newly in default. The regional offices initiate collection and pass on 
the more difficult cases to the private collection agencies. 


2.42 The collection activities of the regional offices have undergone 
significant changes over the past few years as part of the modernization 
strategy. The CRA has trained its regional staff in student loan 
collection procedures, but we found that there are various procedure 
manuals in circulation that have not been consolidated. The Agency 
has also started a project to integrate its Tax and Non-Tax collection 
services: As part of this project, it plans to consolidate policies and 
procedures for the Canada Student Loans Program. 


2.43 We found that the Belleville regional office does a satisfactory job 
of monitoring the performance of collections officers to ensure that 
they comply with procedures. However, while the CRA headquarters 
discusses operational issues with its two regional offices, we noted that 
it does not verify whether monitoring is done properly and consistently 
in both regions. The monitoring relationship between the CRA 
headquarters and the regional offices is not clearly defined. The 
Agency informed us that it is currently designing a formal monitoring 
program for Canada Student Loans collection activities. 


2.44 Recommendation. The Canada Revenue Agency should 
develop and implement a formal quality assurance program to monitor 
whether collection procedures for the Canada Student Loans Program 
are followed properly in its regional offices. 


The Canada Revenue Agency’s response. The Agency agrees with 
this recommendation. As noted by the Office of the Auditor General, 
the Agency has continued to move forward with an extensive 
modernization strategy to improve the recovery of debts owed since it 
assumed responsibility for the collection of defaulted Canada Student 
Loans in August 2005. The strategy already provides for resources to 
be dedicated to the monitoring of collection procedures and 
performance of the Agency’s staff dedicated to this activity. The 
Agency has completed the necessary procedural and operational 
requirements to fully implement a formal monitoring framework, 


commencing the first quarter of the 2007-08 fiscal year. 
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2.45 Monitoring contractors’ compliance with collection 
requirements. The CRA’s contracts with the private collection 
agencies set out their roles and responsibilities. The CRA has also 
developed directives to further define the procedures that collection 
agencies must follow. 


2.46 The CRA and HRSDC have a responsibility to monitor the 
contractors’ compliance with collection requirements in the directives 
and certain clauses of the contracts. We found that the CRA assesses 
the contractors’ performance periodically by looking at amounts 
collected and measures of client service. We also found that HRSDC 
performs periodic reconciliations to ensure that the remittances and 
the account information it receives from the private collection 
agencies match. 


2.417 However, neither the CRA nor HRSDC adequately monitor 
whether the collection activities, including the handling of 
remittances, comply with collection requirements. For example, the 
contracts require annual audits (by audit firms hired by the collection 
agencies) of the adequacy of the agencies’ financial and managerial 
controls for the funds collected. Although these audits would examine 
issues important to the responsibilities of both HRSDC and the CRA, 
we found that neither organization has been reviewing them. 


2.48 Recommendation. The Canada Revenue Agency and Human 
Resources and Social Development Canada should strengthen the 
monitoring of private collection agencies to ensure their compliance 
with contractual and administrative requirements of the Canada 
Student Loans Program. 


Human Resources and Social Development Canada’s response. The 
Department agrees with the recommendation. 


The Canada Revenue Agency (CRA) assumed responsibility for the 
collection of defaulted Canada Student Loans starting in August 2005, 
and thus it is responsible for private collection agencies’ compliance 
with the collection directives. 


Human Resources and Social Development Canada (HRSDC)/Service 
Canada maintained responsibility for establishing overpayments, 
reporting on the outstanding debt for HRSDC clients, recording the 
payments received from the private collection agencies, and ensuring 
client accounts are reconciled. HRSDC/Service Canada will leverage 
the CRA’s efforts toward strengthening the monitoring of the private 
collection agencies and ensure compliance with administrative 
requirements. 
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With respect to the Office of the Auditor General’s observation of the 
private collection agencies’ handling and remittance of accounts, 
HRSDC/Service Canada implemented the Single System of Record 
on 19 January 2007. Payments are now received directly by HRSDC/ 
Service Canada, thus reducing the risk of non-reconciliation of 
account information. HRSDC/Service Canada will closely monitor any 
exceptional payments made to private collection agencies to ensure 
accuracy between remittances and account information. 


The Canada Revenue Agency’s response. The Agency agrees with 
this recommendation. As noted by the Office of the Auditor General, 
the Agency has continued to move forward with an extensive 
modernization strategy to improve the recovery of debts owed since it 
assumed responsibility for the collection of defaulted Canada Student 
Loans in August 2005. The strategy provides for the monitoring of 
private collection agencies in respect of their performance and 
compliance with contractual and administrative requirements. The 
Agency has completed the necessary procedural and operational 
requirements to fully implement a formalized monitoring framework, 
commencing the first quarter of the 2007-08 fiscal year. 


HRSDC cannot assess whether its criteria for Canada Study Grants are being met 


2.49 The regulatory environment. HRSDC has delegated the 
delivery of the Canada Study Grants to the nine provinces and one 
territory participating in the Canada Student Loans Program. They are 
responsible for assessing students’ eligibility, determining the grant 
amount, and disbursing the grants. HRSDC makes annual payments to 
each of the participating governments based on the grant amounts 
each has issued over the three previous years. 


250 The Canada Study Grants are governed by the Canada Student 
Financial Assistance Act and Regulations. The Regulations specify the 
eligibility criteria for each type of Study Grant and the maximum 
amount that can be paid to a qualifying student. The Regulations also 
require general reporting for accountability purposes when HRSDC 
has delegated another party to disburse the grants. 


251 Control structure for the Canada Study Grants. HRSDC’s 
payments to the provinces and territory for Canada Study Grants meet 
the definition of “other transfer payments” as defined in the Treasury 
Board Policy on Transfer Payments. Therefore, we expected HRSDC to 
meet the policy’s objective of ensuring that there is sound 
management, control, and accountability of the transfer payments. 
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Control framework—tThe Canada 
Millennium Scholarship 


Foundation 


2.52 As it has done for Canada Student Loans and Canada Access 
Grants, HRSDC has developed policies and procedures for processing 
the Study Grants and has communicated them to the participating 
jurisdictions. However, the Department currently requires them to 
report only the total amount paid for. Study Grants and the number of 
students receiving each type of Study Grant. HRSDC uses this 
information to allocate the funds for the next fiscal year. Some 
jurisdictions provide more details on grant recipients than others, but 
the data sets are not always complete and the Department cannot 
assess their accuracy. As a result, although the Department can 
account for the payments transferred to the provinces and the 
territory, it cannot assess whether those jurisdictions are awarding the 
correct amounts to the right students. 


2.53 Recommendation. Human Resources and Social Development 
Canada should take appropriate measures to strengthen the control 
structure for the Canada Study Grants to obtain reasonable assurance 
that its grant eligibility criteria are being met. 


Human Resources and Social Development Canada’s response. 
The Department agrees that the Canada Student Loans Program 
would benefit from receiving more complete data and information to 
verify that the participating provinces and the Yukon are properly 
applying the eligibility criteria for Canada Study Grants on behalf of 
the federal government. To improve the accuracy of the control 
structure for the Study Grants, HRSDC will change the conditions 
that the participating provinces and the Yukon must meet in order to 
be eligible for payment of funds awarded as Study Grants. Effective for 
the 2007-08 loan year, the participating provinces and the Yukon will 
have to provide basic personal information (Social Insurance Number- 
level data) for each student awarded a Canada Study Grant in order to 
receive payment. Before the Department approves these payments, it 
will verify the data and will approve payments to the jurisdictions for 
those awards that meet the eligibility criteria. HRSDC recognizes that 
not all provinces have the same degree of preparedness with respect to 
their data systems and will work with each jurisdiction to set 
reasonable timelines for efforts to improve the data. 


2.54 The Canada Millennium Scholarship Foundation has delegated 
the responsibilities to both assess eligibility for its Millennium Bursaries 
and Access Bursaries and distribute them to the provinces and 
territories. Each province and territory is responsible for implementing 
controls to ensure compliance with the Foundation’s requirements. 
The Foundation has an oversight role. 
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2.55 Therefore, we expected the Foundation to have implemented a 
control structure that includes 


* risk assessments, so that controls are designed and adapted based 
on key risks; 


* clear roles and responsibilities; 


¢ clear communication of the Foundation’s requirements, including 
policies and procedures; and 


* ameans to provide the Foundation with reasonable assurance that 
all parties are complying with its requirements. 


The Foundation is assessing the risks of its bursary programs 


“2.56 As permitted by its enabling legislation, the Foundation has 
entered into a separate agreement with each provincial and territorial 
government. It conducts periodic reviews of the provinces’ and 
territories’ compliance with the requirements in the agreements. We 
found that these reviews provide it with sufficient information to 
identify the risks in the process for assessing eligibility and that the 
Foundation takes action on these risks. 


Responsibilities and requirements are clearly defined 


2.51 The Foundation’s agreements with the provinces and territories 
clearly outline the roles and responsibilities of each party and the 
eligibility requirements that students must meet pursuant to the 
Foundation’s enabling legislation. Detailed procedures on specific 
matters, such as data transfers, have also been developed. 


Controls for eligibility assessments and disbursements are appropriate 


2.58 Eligibility. We found that the Foundation has implemented 
adequate controls to ensure that eligibility for Millennium Bursaries 


and Access Bursaries is assessed correctly. 


2.59 To be eligible for a bursary, students must first be approved for 
financial assistance by their province or territory of residence and then 
meet the Foundation’s additional criteria. The Foundation’s periodic 
reviews examine compliance with these criteria. (Quebec is an 
exception: It provides the Foundation with the provincial Auditor 
General’s annual report, which includes information on whether that 
province is complying with the terms of its agreement.) To date, all 
provinces have been reviewed at least once. The results of these 
reviews are presented to the Foundation’s Audit Committee and sent 
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Canada Student Loans Program 


by the Foundation to the Deputy Minister of Education of each 
province and territory. 


2.60 The Foundation has also implemented controls to validate the 
information it receives from each province and territory in order to 
process the bursaries. These controls provide reasonable assurance that 
the information is complete, accurate, and valid. 


2.61 Disbursements. The Foundation is responsible for approving, 
disbursing, and recording the bursaries. It issues the cheques directly to 
students in the Yukon, Northwest Territories, and Ontario. The 
Foundation sends the other provinces and Nunavut lump sums in trust 
for the bursary recipients to be paid directly to the students or applied 
to reduce their outstanding debt. We found that the Foundation has 
implemented key controls to ensure that the disbursement process is 
correct: It validates and reconciles the data for disbursements to 
ensure that it is complete and accurate, and its Board of Directors 
approves all payments. 


2.62 The Canada Student Loans Program is intended to promote 
access to post-secondary education for students in financial need. To 
achieve this, the Program has set four goals: 


° increase awareness; 
¢ make the loan experience a positive one; 
* ensure performance, integrity, and accountability; and 


* maintain the government’s commitment to access. 


2.63 We expected Human Resources and Social Development 
Canada to have satisfactory procedures in place to 


¢ collect the information necessary for measuring performance 
against the Program’s goals; 


¢ make the information available for decision makers to use for 
improving results; and 


* report against objectives, including clear performance 
expectations, key results against expectations, and clear 
explanations for results that fall short of expectations. 


HRSDC is taking steps to increase awareness of the Program 


2.64 The first goal of the Canada Student Loans Program is to 
increase awareness of the Program. We found that HRSDC has 
gathered information that has enabled it to identify gaps in awareness 
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of the Program among potential post-secondary students and their 
families. We also found that the Department has acted on its findings. 


2.65 Specifically, HRSDC conducted research and surveys 

between 2004 and 2006 that indicated a lack of awareness about the 
federal financial assistance programs for post-secondary students. The 
Department now works with the provinces and the territory to 
promote the Program. Examples of its efforts include the following: 


* a pan-Canadian five-year communication campaign, begun 
in 2005, to encourage parents to plan and save for their children’s 
education; 


* an upgrade to the “CanLearn” website, which explains how to 
apply for, maintain, and repay federal and provincial or territorial 
student loans; and 


* a pilot project (2005-06) in one province for the Canada Student 
Loans Program to test peer-to-peer communication. In the 
project, students were trained to make presentations at high 
schools; the Department plans to expand the project to another 
province in 2007. 


2.66 HRSDC also conducted qualitative research to refine its 
communication material and media approach for the Program. As a 
result, it designed a more concise version of its information guide. 


Client satisfaction with the Program is near target 


2.67 The second goal of the Canada Student Loans Program is to 
make the loan experience a positive one. 


2.68 HRSDC conducts annual surveys of borrowers to measure their 
satisfaction with the Program’s service delivery. According to these 
surveys, satisfaction levels have ranged between 63 percent and 

75 percent since 2002. For 2006-07, the Department has set a target 
of 76 percent. 


2.69 HRSDC offers a payment incentive to the service providers 
based in part on survey results that exceed set performance thresholds. 
In the 2004 to 2006 surveys, overall borrower satisfaction ratings met 
or exceeded these thresholds. Borrowers gave higher ratings to the 
service providers on several elements of client service, such as accuracy 
of information, courteousness, and receipt of communications in the 
official language of choice. However, they gave lower ratings to the 
service providers for the time callers were waiting on the telephone for 


service and the extent to which problems were resolved to the 


Report of the Auditor General of Canada—May 2007 


Chapter 2 | 19 


FEDERAL LOANS AND GRANTS FOR POST-SECONDARY EDUCATION— 
HUMAN RESOURCES AND SOCIAL DEVELOPMENT CANADA AND CANADA MILLENNIUM SCHOLARSHIP FOUNDATION 


Annual default rate—The percentage of loans 
in repayment that are turned over to the Canada 
Revenue Agency for collection during a year. 


borrowers’ satisfaction. HRSDC’s new contract with a single service 
provider contains some new service incentives, including a shorter 
standard response time from the call centres and a flexible incentive 
scheme that can be fine-tuned as part of a process that is renewed 
annually. 


The Program’s Portfolio Management Strategy is well managed and achieving 
results 


2.70 The third goal of the Canada Student Loans Program is to 
ensure the Program’s performance, integrity, and accountability. 


271 As already noted (see paragraph 2.35), HRSDC uses a number of 
ways to ensure program integrity and accountability. Reducing the 
number of defaulted loans is another important part of achieving the 
Program’s third goal. Early in the shift to providing loans directly 
instead of through financial institutions, HRSDC developed its 
Portfolio Management Strategy as a response to what it perceived as 
high default rates. The strategy’s main goals are to reduce loan default 
rates and improve program integrity through sound debt management 
and good communication with borrowers. 


2.72 Program officials meet monthly with the service providers to 
discuss the performance and integrity of the portfolio and to identify 
trends and emerging challenges. HRSDC regularly reviews reports 
from the service providers about the portfolio, and about their 
performance against contract obligations and use of debt management 
measures. 


2.13 The Department has improved communication with Program 
borrowers who are repaying their loans, and it continues to work with 
service providers to improve their communication with high-risk 
borrowers. It has implemented measures proposed by the service 
providers, based on their experience in loan management. These 
measures include conducting risk profiles of borrowers, maintaining 
regular contact with borrowers, and enhancing the tracing of 
delinquent borrowers. The Department measures the results of 
these initiatives through the monthly performance reports it receives 
from the service providers. Annual default rates decreased from 
around 22 percent to 9 percent between 2002 and 2006. 


2.74 Despite improvements in the default rate, the Department's 
analysis shows that students lack awareness about the Program’s debt 
management measures. HRSDC has conducted studies to find out 
more about borrowers who have defaulted and about those who use 
interest relief. In 2005, it set up focus groups on interest relief; in 2006, 
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together with Statistics Canada, it published a study on the use of the 
Interest Relief Program. The study noted that this program was used by 
only 45 percent of eligible borrowers in the repayment phase. To 
increase the use of debt management measures, the Department has 
drafted a communication plan and raised the income threshold for 
interest relief eligibility; it is also considering other actions. 


HRSDC needs to measure the Program’s performance on access 


215 The fourth goal of the Program is to maintain the government’s 
commitment to making post-secondary education more accessible to 
Canadians. 


2.16 We found that HRSDC does not have a sound performance 
measurement strategy for access—that is, a means to select and 
develop measures for ongoing use in decision making. Without a clear 
strategy, it cannot effectively and consistently measure and report on 
whether the Program is achieving the access goal. 


2.717 In 2003, HRSDC identified a number of performance indicators 
on access. However, it has not developed a systematic approach to 
collecting appropriate information and putting it to use. 


2.18 The Department collects some performance data from its 
program partners. However, they often submit the data in different 
formats, with different elements, and at different times. The 
Department told us that the information is insufficient to meet its 
needs and that it is working to harmonize and simplify the systems and 
practices to fix this. Even where the Department does have provisions 
for exchanging data with provinces—for example, as part of the 
integration agreements (see paragraph 2.9)—these provisions are not 
used. Moreover, HRSDC has no single repository for the data that it 
collects and no business process rules for managing the data. 


2.79 In 2001, HRSDC committed to completing a summative 
evaluation of the Canada Student Loans Program in 2006. The 
Department began planning the evaluation in 2006, approaching it as 
a series of studies to be completed by 2011. Officials told us that this 
approach would be included in the departmental five-year evaluation 
plan, which was scheduled for approval early in 2007. At the time of 
the audit, plans had been clearly defined for only the first two years 
and consisted mainly of feasibility and concept studies. Without a more 
precise framework, the evaluation risks losing focus while being further 


delayed. 
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Three-year loan default rate—The percentage 
of loans for which repayment is due to begin in a 
particular year and that are turned over to the 
Canada Revenue Agency for collection within 
three years. 


280 Recommendation. Human Resources and Social Development 
Canada should develop a sound performance measurement strategy for 
the Canada Student Loans Program’s goal to make post-secondary 
education more accessible to Canadians. 


Human Resources and Social Development Canada’s response. 
The Department has a performance measurement strategy for the 
Canada Student Loans Program and reports on the goal of making 
post-secondary education more accessible to Canadians. It reports on 
program performance through the Program’s Annual Report, the 
Departmental Performance Report, the Report on Plans and Priorities, 
and its website. The Department will better articulate this 
performance measurement strategy in a single framework document. 


2.81 Recommendation. Human Resources and Social Development 
Canada should complete a summative evaluation that focuses on 
program impacts and outcomes and should do so within the five years 
specified in its plans. 


Human Resources and Social Development Canada’s response. 
Departmental management has adopted a five-year phased approach 
to the summative evaluation of the impacts and outcomes of the 
Canada Student Loans Program. The evaluation is under way, with the 
first synthesis report of the evaluative work available in 2006-07. 
Annual synthesis reports will be produced, culminating in a final 
summary report on program outcomes, which will be available in 2011. 


Parliament does not have a full picture of the Program’s performance 


2.82 We found that the Department’s reporting on program results is 
fragmented; some significant matters are reported in the Departmental 
Performance Report and others are stated in the Canada Student 
Loans Program Annual Report. 


2.83 In its 2005-06 Departmental Performance Report, HRSDC 
reported on three indicators for the Canada Student Loans Program: 
the client satisfaction level, the three-year loan default rate, and the 
number of students receiving federal loans for post-secondary 
education. However, these indicators were not clearly linked to the 
Program’s strategic goals. Moreover, the Department did not report on 
whether it had increased awareness of the Program or had improved 
access to post-secondary education. In part, this reflects the absence of 
a sound performance measurement strategy for the access goal (see 
paragraph 2.76). 
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2.84 The Canada Student Financial Assistance Act requires the 
Department to table an annual report to Parliament on the Canada 
Student Loans Program. The 2003-04 Annual Report gives details on 
program activities and outputs—for example, the number of student 
loans given in each province and territory. However, it does not report 
against targets or specify whether the Program is meeting its goals. 
Further, the Department tabled the Program’s 2003-04 Annual Report 
in August 2006; at the time of our audit, it had not yet tabled 

the 2004-05 Annual Report. 


2.85 We noted that in its 2006-07 Report on Plans and Priorities, the 
Department set expectations for only two of its four strategic goals— 
making the loan experience a positive one and improving performance, 
integrity, and accountability. It did not set targets for increasing 
awareness of the Program or improving access to post-secondary 
education. 


2.86 Through the arrangement for conditional grants, HRSDC is 
required to include in its Report on Plans and Priorities and 
Departmental Performance Report significant expected results from 
the Canada Millennium Scholarship Foundation and to situate these 
within the Department's overall results. Its 2004-05 Departmental 
Performance Report provided timely information on the Foundation’s 
activities and outputs but did not explain how they related to the 
Department’s own results. The 2005—06 Departmental Performance 
Report has minimal information about the Foundation. 


2.81 Recommendation. In its Departmental Performance Report, 
Human Resources and Social Development Canada should 
consolidate and improve its reporting on key indicators of progress 
toward the strategic goals of the Canada Student Loans Program and 
clarify how the activities of the Canada Millennium Scholarship 
Foundation relate to its own activities and goals. It should also provide 
for timelier tabling of the Canada Student Loans Program Annual 


Report. 


Human Resources and Social Development Canada’s response. 

As noted in the Department’s response to recommendation 2.80, the 
Department will articulate its performance measurement strategy for 
the Canada Student Loans Program in one framework document. This 
will consolidate all existing and potential key indicators for reporting 
on the progress of the Program. It should be noted that availability of 
data for the potential indicators is a challenge as not all provinces have 
the same degree of preparedness for their data systems. However, the 


Department will work to ensure that these key indicators will continue 
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Foundation 


to be monitored appropriately and results will be reported when 
available through future departmental performance reports. 
Furthermore, the Department will report on the relationship between 
the Canada Millennium Scholarship Foundation’s activities and its 
own through its departmental performance reports, while recognizing 
the arms-length status of the Foundation. HRSDC will improve the 
time frame within which it tables the Canada Student Loans Program 
Annual Report each year. 


2.88 The Foundation’s main objective is to improve access to 
post-secondary education. It is committed to 


* increasing the visibility of its programs, 
¢ delivering bursaries to eligible students, and 


* improving its understanding of issues related to access. 


2.89 We expected the Foundation to have satisfactory procedures in 
place to 


¢ collect the information necessary for measuring performance 
against objectives; 


¢ make the information available for decision makers to use in 
improving results; and 


* report against objectives, including clear performance 
expectations, key results against expectations, and clear 
explanations for results that fall short of expectations. 


The Foundation collects and uses performance information 


2.90 Visibility of the programs. We found that the Foundation has 
been gathering relevant information on awareness of its programs. It is 
using this information to increase their visibility. 


2.91 For example, the Foundation conducted a series of surveys 
between 2001 and 2003 on financial assistance programs for 
post-secondary education, and in 2004 it held broad-based 
consultations with students, parents, and education professionals. 
These surveys and consultations showed that the Foundation lacked 
visibility. During our audit, the Foundation was conducting a survey on 
awareness of its own programs and planned to publish the results early 
in 2007. It was also seeking to collaborate with HRSDC and the 
provinces and territories to produce a new pamphlet with specific 
information on its Millennium and Access Bursaries. 
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2.92 The Foundation’s research showed that awareness efforts should 
focus on high school students who often get information about 
financial assistance through peers, parents, teachers, counsellors, and 
others. Therefore, the Foundation has launched three pilot projects in 
high schools to increase awareness of and participation in 
post-secondary education. It is also targeting information to parents 
and others who interact with high school students. 


2.93 Perceptions of services in delivering bursaries. The 
Foundation has little interaction with recipients of the Millennium and 
Access Bursaries and currently has limited information about their 
perceptions of its services. To fill this gap, the survey on awareness also 
sought Millennium Bursary recipients’ perception of the Foundation’s 
services. 


2.94 Understanding of access issues. The Foundation needs to 
gather and assess data from provinces and territories to measure its 
programs’ results and impacts. The Foundation’s agreements with 
provinces and territories specify what data they are to provide about 
award recipients and when they should send the data. We found that 
the Foundation’s systems and practices are adequate to ensure that it 
collects good-quality data in a timely manner. 


2.95 The Foundation’s research program provides additional 
information about the results of its bursary programs. The research 
aims to enhance understanding of barriers to post-secondary education 
and to provide data and analysis to monitor the impact of the 
Foundation’s bursary programs. Research reports are shared with 
decision makers and the public. 


296 From research and its 2004 consultations, the Foundation found 
that certain groups were under-represented in the post-secondary 
student population and that it needed to expand its program coverage. 
In 2005 it created its Access Bursary program to encourage 
low-income, rural, and Aboriginal students to continue their 
education. At the time of the audit, it was conducting an initial 
assessment of this program. The Foundation plans to complete an 
evaluation of its awareness efforts and Access Bursary program 


by 2009. 


291 The Foundation’s enabling legislation stipulates that it is to grant 
scholarships in a manner that complements provincial and territorial 
financial assistance programs for post-secondary students. When the 
Foundation initiated its Millennium Bursary program in 1999, it 


conducted assessments of the provincial and territorial programs. It did 
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the Foundation 


so again in 2004 and most recently in November 2006. It has also done 
further assessments of provinces’ and territories’ programs as needed. 
The Foundation has concluded that its own program complements 
those of the provinces and territories. 


The Foundation provides the required information to Parliament 


2.98 The Foundation’s enabling legislation requires it to provide the 
Minister of Human Resources and Social Development with an annual 
report on its performance, including audited financial statements. The 
government also requires (in Budget 2003 commitments) that the 
Foundation provide summaries of its corporate plans to Parliament. We 
found that the Foundation is meeting these requirements. While its 
past reports only described activities, its more recent annual reports 
(2004 and 2005) provide additional details about the Foundation’s 


impact on reducing student debt and increasing student funds. 


HRSDC and the Foundation need to improve arrangements for consulting together 


2.99 Human Resources and Social Development Canada and the 
Canada Millennium Scholarship Foundation are working toward the 
same public policy objective—improved access to post-secondary 
education. In our 2005 February Status Report, Chapter 4, 
Accountability of Foundations, we said that strategic monitoring of 
foundations by sponsoring departments, in the form of contact and 
consultations, is a good practice for sound delivery of shared public 
policy. In this audit, we found that HRSDC and the Foundation could 
improve their exchange of information on issues of mutual interest, 
which would help achieve their shared objective. 


2.100 For example, they do not always exchange information that 
would be useful for the delivery of shared public policy. Under its 
agreements with provinces and the territories, the Foundation 
conducts regular reviews in most jurisdictions to verify that its 
eligibility criteria for bursaries have been met. Although these reviews 
focus on the Foundation’s criteria, HRSDC’s loan and grant programs 
share some of the same criteria. For example, recipients of both the 
Foundation’s and the Department’s bursaries, grants, and loans must 
be permanent residents of Canada. However, we found that audit 


findings that would be of interest to HRSDC are not necessarily 
shared. 


2.101 In another case, the Department and the Foundation 
communicated on program issues, but the exchange was not 
well documented. Responding to research findings that some 
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populations were not being reached by federal, provincial, and 
territorial student financial assistance programs, HRSDC set up the 
Canada Access Grant program and the Foundation set up its Access 
Bursary program. The eligibility criteria of these programs are similar. 
To avoid duplication or overlap, officials from both organizations 
communicated informally before implementing the programs. The 
Department’s and the Foundation’s programs are now complementary. 
However, little of the content of these consultations and their 
outcomes was documented. It is important that the two organizations 
improve their arrangements for consulting on any future program 
changes to ensure that they continue to be well coordinated. It is also 
important to document these exchanges to aid future decision making. 


2.102 We found that HRSDC and the Foundation have developed a 
good research relationship. They co-fund a number of research 
projects aimed at gaining a better understanding of student financial 
assistance and its impacts on access to post-secondary education. The 
Department told us that over the last year, there have been more 
frequent discussions of research and improved consultations with the 
Foundation on program and policy issues. 


2.103 Recommendation. Human Resources and Social Development 
Canada and the Canada Millennium Scholarship Foundation should 
improve their arrangements for consulting one another on issues of 
mutual interest in order to achieve their shared objective. The two 
organizations should document their consultations and the outcomes 
to aid future decision making. 


Human Resources and Social Development Canada’s response. 
Officials of the Canada Student Loans Program consult with officials of 
the Canada Millennium Scholarship Foundation regularly on areas of 
shared program objectives while recognizing the arms-length status of 
the Foundation. The Department will institute procedures to ensure 
that results of key discussions with the Foundation are recorded to aid 
future decision making for areas of shared program outcomes or policy 
development interests. In support of improved consultation, HRSDC 
has invited the Foundation to join the National Advisory Group on 
Student Financial Assistance, which will meet twice annually starting 
in March 2007. The results of the National Advisory Group's 
consultations are documented in formal minutes and records of 


decision. 


The Canada Millennium Scholarship Foundation’s response. The 
Canada Millennium Scholarship Foundation looks forward to 
continuing its consultation with Human Resources and Social 
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Development Canada (HRSDC) in the future and will document 
exchanges of information when appropriate. Where improved 
arrangements are required, it is the Foundation’s view that these 
arrangements acknowledge the importance of both organizations 
consulting not only each other but also the provinces and territories. 
In this regard, the Foundation welcomes HRSDC’s recent invitation to 
join the National Advisory Group on Student Financial Assistance, a 
group that includes among its members both the federal and the 
provincial co-chairs of the Intergovernmental Consultative 
Committee on Student Financial Assistance. The Committee serves as 
the federal-provincial multilateral forum for consultation on policy and 
program development on student assistance. The Foundation will send 
a representative to meetings of the National Advisory Group beginning 
in the spring of 2007. In addition, the Foundation believes it would be 
useful to be invited periodically to attend meetings of the 
Intergovernmental Consultative Committee. The Foundation would 
welcome the opportunity to appear before the Committee annually or 
biannually to present an update on its operations and plans, to review 
issues raised in audits and other reviews, and to consult with its federal, 
provincial, and territorial colleagues on issues of mutual interest. 


Conclusion 


2.104 With a few exceptions, the Canada Student Loans and the 
Canada Access Grant programs are well managed and have the 
necessary controls in place. 


2.105 HRSDC has appropriate controls in place to ensure that its 
partners in the delivery of Canada Student Loans and Access Grants 
are exercising due care in their responsibilities. However, the 
Department has not clearly documented the information that it 
gathers on the adequacy of controls for assessing eligibility for its loans 
and Access Grants. HRSDC’s control structure for the Canada Study 
Grants does not provide the Department with adequate assurance that 
its eligibility criteria are being met. 


2.106 The Canada Revenue Agency monitors the performance of 
private collection agencies, and Human Resources and Social 
Development Canada monitors the consistency of information 
provided by the agencies. However, neither organization adequately 
monitors other key collection requirements of the Canada Student 
Loans Program. 
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2.107 HRSDC is taking steps to enhance awareness of its loan and 
grant programs and of the measures available to help students manage 
debt. However, lack of a clear performance measurement strategy has 
contributed to incomplete reporting of the results of the Canada 
Student Loans Program. The Department has postponed the program 
evaluation that it had committed to doing by 2006. It has not yet fully 
defined the framework for the evaluation, thus risking further delays. 
Consequently, is has not yet given Parliament a full picture of the 
Program’s performance. 


2.108 The Canada Millennium Scholarship Foundation’s bursary 
programs are well managed. The Foundation has the necessary 
controls in place to ensure that bursaries are distributed correctly. It 
also obtains reasonable assurance that the provinces and territories are 
fulfilling their responsibilities in delivering the programs. 


2.109 The Foundation is taking steps to improve general awareness of 
its Millennium Bursaries. It collects adequate performance information 
and meets its obligation to report the results of its activities to 
Parliament. It has used this information together with its research 
findings to try to reach more students through its Access Bursaries. 


2.110 HRSDC and the Foundation are both working to improve access 
to post-secondary education. Improving their arrangements for 
consulting on issues of mutual interest would help achieve this shared 
objective. 
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About the Audit 


Objective 


The audit objective was to determine whether federal loan and debt reduction programs for 
post-secondary students are managed to improve access to post-secondary education while ensuring that 
the necessary controls are in place. 


Scope and approach 
The scope of the audit included 


- the Canada Student Loans Program—the loans program under the direct loans regime, and the 
Canada Access and Study Grants; and 


- the Millennium Bursary and Millennium Access Bursary programs of the Canada Millennium 
Scholarship Foundation. 


The main entities involved in the audit were Human Resources and Social Development Canada, the 
Canada Revenue Agency (CRA), and the Canada Millennium Scholarship Foundation. We examined 
practices at the national headquarters of each organization and visited one of the two CRA regional offices 
responsible for loan collections. 


We conducted interviews, document reviews, data analyses, and procedural walk-throughs to assess 
whether the entities were meeting our audit criteria. We also reviewed documents prepared by audit firms 
engaged by HRSDC and the Foundation and interviewed some of the firms’ personnel to ensure that we 
could use their work as key controls. 


In addition, we reviewed documents of other federal organizations that contribute to delivering federal 
post-secondary loan and grant programs: the Department of Finance, Public Works and Government 
Services Canada, and the Treasury Board Secretariat. 


We conducted audit work on the current program of direct loans but not on repayments due under the 
former program arrangements under which private-sector financial institutions gave loans to students. 


We did not audit any other federal programs that support post-secondary education; nor did we conduct 
audit work in provinces and territories or at any of the private sector agencies that provide program 
services. 


We did not audit the Foundation’s merit awards programs or its research activities. 


In the contracts signed between parties involved in the loan and grant programs, we examined only 
the terms that were relevant to delivering the programs and ensuring that the necessary controls are in 
place. We did not examine how the contracts were awarded or managed. 
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Criteria 
The criteria for the audit were based on 
- the results management framework for the Government of Canada, 
+ the Treasury Board Secretariat’s guides for performance evaluation and reporting, 
* our Office’s recent reports on results measurement and reporting, 
- the Treasury Board Policy on Transfer Payments, and 


* recognized standards of sound business and financial practices. 


The criteria were the following: 


- Appropriate controls are in place to ensure that programs are delivered in compliance with legislation, 
policy, agreements, and contracts. 


- Information on programs is easy to find and understand, and services are provided in ways that are 
accessible and convenient. 


* Satisfactory procedures are in place to collect and use performance information to manage programs 
and to report on program results. 


Audit work completed 


Audit work for this chapter was substantially completed on 30 November 2006. 


Audit team 


Assistant Auditor General: Nancy Cheng 
Principal: Sylvain Ricard 
Directors: Nicholas Swales and Esther Becker 


Kasia Borowska 
Lucie Cardinal 
Véronique Duguay 
Annik Leblanc 
Marc Legrand 
TinaLise LeGresley 
Felix Meunier 
Stephanie Moores 
Julie Salois 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 2. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


2.38 ‘To ensure that the knowledge 
acquired about the provinces’ and the 
territory’s processes for eligibility and 
needs assessments is available for 
managing the Canada Student Loans 
Program, Human Resources and Social 
Development Canada should document 
its conclusions on the adequacy of 
controls and any related action taken. 


(2.36-2.37) 


2.44. The Canada Revenue Agency 
should develop and implement a formal 
quality assurance program to monitor 
whether collection procedures for the 
Canada Student Loans Program are 


followed properly in its regional offices. 
(2.39-2.43) 


The Department agrees that the Canada Student Loans Program 
would benefit from better documentation of provincial and 
territorial processes for eligibility and needs assessment. Program 
officials already meet regularly with officials from participating 
jurisdictions to share information on their respective controls of 
these processes. The Department will institute procedures to 


ensure that its conclusions on the adequacy of the Program’s 
controls for eligibility and needs assessments are documented. 


The Agency agrees with this recommendation. As noted by the 
Office of the Auditor General, the Agency has continued to 
move forward with an extensive modernization strategy to 
improve the recovery of debts owed since it assumed 
responsibility for the collection of defaulted Canada Student 
Loans in August 2005. The strategy already provides for 
resources to be dedicated to the monitoring of collection 
procedures and performance of the Agency’s staff dedicated to 
this activity. The Agency has completed the necessary 
procedural and operational requirements to fully implement a 
formal monitoring framework, commencing the first quarter of 


the 2007-08 fiscal year. 
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2.48 The Canada Revenue Agency 
and Human Resources and Social 
Development Canada should 
strengthen the monitoring of private 
collection agencies to ensure their 
compliance with contractual and 
administrative requirements of the 
Canada Student Loans Program. 


(2.45-2.47) 
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Human Resources and Social Development Canada’s response. 
The Department agrees with the recommendation. 


The Canada Revenue Agency (CRA) assumed responsibility for 
the collection of defaulted Canada Student Loans starting in 
August 2005, and thus it is responsible for private collection 
agencies’ compliance with the collection directives. 


Human Resources and Social Development Canada (HRSDC)/ 
Service Canada maintained responsibility for establishing 
overpayments, reporting on the outstanding debt for HRSDC 
clients, recording the payments received from the private 
collection agencies, and ensuring client accounts are reconciled. 
HRSDC/Service Canada will leverage the CRA’s efforts toward 
strengthening the monitoring of the private collection agencies 
and ensure compliance with administrative requirements. 


With respect to the Office of the Auditor General’s observation 
of the private collection agencies’ handling and remittance of 
accounts, HRSDC/Service Canada implemented the Single 
System of Record on 19 January 2007. Payments are now 
received directly by HRSDC/Service Canada, thus reducing the 
risk of non-reconciliation of account information. HRSDC/ 
Service Canada will closely monitor any exceptional payments 
made to private collection agencies to ensure accuracy between 
remittances and account information. 


The Canada Revenue Agency’s response. The Agency agrees 
with this recommendation. As noted by the Office of the 
Auditor General, the Agency has continued to move forward 
with an extensive modernization strategy to improve the 
recovery of debts owed since it assumed responsibility for the 
collection of defaulted Canada Student Loans in August 2005. 
The strategy provides for the monitoring of private collection 
agencies in respect of their performance and compliance with 
contractual and administrative requirements. The Agency has 
completed the necessary procedural and operational 
requirements to fully implement a formalized monitoring 
framework, commencing the first quarter of the 2007-08 
fiscal year. 
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2.53 Human Resources and Social 
Development Canada should take 
appropriate measures to strengthen the 
control structure for the Canada Study 
Grants to obtain reasonable assurance 
that its grant eligibility criteria are being 
met. (2.49-2.52) 


The Department agrees that the Canada Student Loans Program 
would benefit from receiving more complete data and 
information to verify that the participating provinces and the 
Yukon are properly applying the eligibility criteria for Canada 
Study Grants on behalf of the federal government. To improve 
the accuracy of the control structure for the Study Grants, 
HRSDC will change the conditions that the participating 
provinces and the Yukon must meet in order to be eligible for 
payment of funds awarded as Study Grants. Effective for the 
2007-08 loan year, the participating provinces and the Yukon 
will have to provide basic personal information (Social Insurance 
Number-level data) for each student awarded a Canada Study 
Grant in order to receive payment. Before the Department 
approves these payments, it will verify the data and will approve 
payments to the jurisdictions for those awards that meet the 
eligibility criteria. HRSDC recognizes that not all provinces have 
the same degree of preparedness with respect to their data 
systems and will work with each jurisdiction to set reasonable 
timelines for efforts to improve the data. 


Performance management—Canada Student Loans Program 


2.80 Human Resources and Social 
Development Canada should develop a 
sound performance measurement 
strategy for the Canada Student Loans 
Program’s goal to make post-secondary 
education more accessible to 


Canadians. (2.75—2.78) 


2.81 Human Resources and Social 
Development Canada should complete 
a summative evaluation that focuses on 
program impacts and outcomes and 
should do so within the five years 
specified in its plans. (2.79) 


The Department has a performance measurement strategy for 
the Canada Student Loans Program and reports on the goal of 
making post-secondary education more accessible to Canadians. 
It reports on program performance through the Program’s 
Annual Report, the Departmental Performance Report, the 
Report on Plans and Priorities, and its website. The Department 
will better articulate this performance measurement strategy in a 
single framework document. 


Departmental management has adopted a five-year phased 
approach to the summative evaluation of the impacts and 
outcomes of the Canada Student Loans Program. The 
evaluation is under way, with the first synthesis report of the 
evaluative work available in 2006-07. Annual synthesis reports 
will be produced, culminating in a final summary report on 
program outcomes, which will be available in 2011. 
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2.87 In its Departmental Performance 
Report, Human Resources and Social 
Development Canada should 
consolidate and improve its reporting 
on key indicators of progress toward the 
strategic goals of the Canada Student 
Loans Program and clarify how the 
activities of the Canada Millennium 
Scholarship Foundation relate to its 
own activities and goals. It should also 
provide for timelier tabling of the 
Canada Student Loans Program 
Annual Report. (2.82-2.86) 


Consultation between HRSDC and the Foundation 


2.103 Human Resources and Social 
Development Canada and the Canada 
Millennium Scholarship Foundation 
should improve their arrangements for 
consulting one another on issues of 
mutual interest in order to achieve their 
shared objective. The two organizations 
should document their consultations 
and the outcomes to aid future decision 


making. (2.99-2.102) 
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As noted in the Department’s response to recommendation 2.80, 
the Department will articulate its performance measurement 
strategy for the Canada Student Loans Program in one 
framework document. This will consolidate all existing and 
potential key indicators for reporting on the progress of the 
Program. It should be noted that availability of data for the 
potential indicators is a challenge as not all provinces have the 
same degree of preparedness for their data systems. However, the 
Department will work to ensure that these key indicators will 
continue to be monitored appropriately and results will be 
reported when available through future departmental 
performance reports. Furthermore, the Department will report 
on the relationship between the Canada Millennium Scholarship 
Foundation’s activities and its own through its departmental 
performance reports, while recognizing the arms-length status of 
the Foundation. HRSDC will improve the time frame within 
which it tables the Canada Student Loans Program Annual 
Report each year. 


Human Resources and Social Development Canada’s response. 
Officials of the Canada Student Loans Program consult with 
officials of the Canada Millennium Scholarship Foundation 
regularly on areas of shared program objectives while recognizing 
the arms-length status of the Foundation. The Department will 
institute procedures to ensure that results of key discussions with 
the Foundation are recorded to aid future decision making for 
areas of shared program outcomes or policy development interests. 
In support of improved consultation, HRSDC has invited the 
Foundation to join the National Advisory Group on Student 
Financial Assistance, which will meet twice annually starting 

in March 2007. The results of the National Advisory Group's 
consultations are documented in formal minutes and records 

of decision. 


The Canada Millennium Scholarship Foundation’s response. 
The Canada Millennium Scholarship Foundation looks forward 
to continuing its consultation with Human Resources and Social 
Development Canada (HRSDC) in the future and will 
document exchanges of information when appropriate. Where 
improved arrangements are required, it is the Foundation’s view 
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that these arrangements acknowledge the importance of both 
organizations consulting not only each other but also the 
provinces and territories. In this regard, the Foundation 
welcomes HRSDC’s recent invitation to join the National 
Advisory Group on Student Financial Assistance, a group that 
includes among its members both the federal and the provincial 
co-chairs of the Intergovernmental Consultative Committee on 
Student Financial Assistance. The Committee serves as the 
federal-provincial multilateral forum for consultation on policy 
and program development on student assistance. The 
Foundation will send a representative to meetings of the 
National Advisory Group beginning in the spring of 2007. In 
addition, the Foundation believes it would be useful to be invited 
periodically to attend meetings of the Intergovernmental 
Consultative Committee. The Foundation would welcome the 
opportunity to appear before the Committee annually or 
biannually to present an update on its operations and plans, to 
review issues raised in audits and other reviews, and to consult 
with its federal, provincial, and territorial colleagues on issues of 
mutual interest. 
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Canada 


Main Points 


The Department of Foreign Affairs and International Trade 
operates in a highly challenging and complex environment that 
requires it to respond quickly to changing local and international 
circumstances. It has staff based in Canada and in over 170 missions 
abroad, where it also employs a culturally diverse local workforce of 
over 5,000 employees who are subject to various local labour laws. 
In the 2005-06 fiscal year, the Department spent $830 million 

on salaries and benefits. 


We examined whether the Department plans for and manages its 
human resources in a way to ensure that it has the people with the 
right skills and competencies to carry out its mandate. Our audit 
included interviews, file reviews, and data analysis at the Department’s 
headquarters in Ottawa and at 12 of its missions abroad. 


Canada relies on the people who work in its foreign missions to be its 
“face abroad.” They represent Canada in such fields as political, trade 
and business relations, diplomacy, culture, and human rights. 
Canadians travelling abroad depend on the missions for a range of 
assistance, from replacing lost passports to determining their 
whereabouts, and seeing to their safety in emergency situations. The 
government relies on the Department for timely, high-quality advice 
on international issues to help it formulate foreign policy. The 
Department also has a position of leadership toward other government 
departments operating abroad to ensure a unified Canadian approach. 


It is important to Canadians that the Department have the right 
people in the right place at the right time and that it be able to act 
quickly and decisively in a rapidly changing environment. Given the 
complexity of its workforce, its international role, and the demographic 
challenge it faces, strategic planning and management of its human 


resources are critical to achieving its mandate. 
- The Department has no strategic human resources plan. It does not 


have a complete picture of the people, competencies, and experience 


it will need in the future, and it lacks basic information needed to 
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plan for and manage its workforce. For example, it does not have 
accurate information on the number of vacant positions it has to fill. 
Until recently, the Department has paid little attention to human 
resources planning. Since the 2005-06 fiscal year, it has engaged in a 
major effort to develop a strategic planning framework for managing 
its people, but much remains to be done. 


The Department has been unable through its recruitment, 
promotion, and assignment processes to fill its positions on a timely 
basis with enough people who have all the required skills and 
competencies. Fifty-eight per cent of its employees in the 
management category will be eligible to retire by 2010; for all 
categories combined, the figure is 26 percent. Already, departures 
now outnumber new hires. These are strong indications that the 
Department is at risk of not having the human resources it will need 
to carry out its mandate effectively. 


The process in place to compensate employees for living conditions 
abroad and provide incentives for hardship postings has not allowed 
the Department to respond in a timely way to changing 
circumstances and the problems employees face as a result. This 
hinders the Department’s ability to assign qualified staff to some of 
its missions around the world. Despite concerns expressed over many 
years by our Office and various stakeholders, this issue has not been 
resolved. 


The management of locally engaged staff, who constitute half of the 
Department’s workforce, gets little attention. The Department does 
not have all the information it needs to properly manage these 
employees and is not adequately supporting its missions in their 
human resources management responsibilities. 


The Department of Foreign Affairs and International Trade and the 
Treasury Board Secretariat have responded. The Department of 


Foreign Affairs and International Trade and the Treasury Board 


Secretariat have agreed with our recommendations. Their detailed 


responses follow the recommendations throughout the chapter. 
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Introduction 


The Department’s role 


3.1 The Department of Foreign Affairs and International Trade 
delivers on Canada’s international agenda by carrying out four key 
functions: 


* leading and coordinating international policy-making for the 
Government of Canada; 


* promoting Canadian interests, values, and expertise abroad; 


* assisting Canadian business and the public by providing trade, 
investment, consular, and passport services; and 


* providing the facilities and administrative infrastructure outside 
Canada that enables the global operations of the entire federal 
government and the provincial governments. 


3.2 In 2005 the Government of Canada tabled its International 
Policy Statement, which, among other responsibilities, gave the 
Department a clear role to play in coordinating Canada’s efforts on the 
international scene. One of the policy’s key priorities is to enhance 
Canada’s relationship with the United States, and to broaden and 
deepen its ties with rising powers such as China, India, and Brazil. 


3.3. The Department is headquartered in Ottawa. It operates about 
170 missions in 111 countries around the world to represent the 
Government of Canada and advance Canadian interests bilaterally 
with other countries or multilaterally with international organizations. 


Human resources structure 


3.4 The Department is organized into two operational and eight 
functional branches, as well as two administrative branches, one of 
which is dedicated to supporting the management of human resources 
(Human Resources Branch). For each mission abroad, there is a head of 
mission (such as an ambassador) who is responsible for the management, 
direction, and supervision of the official activities of the various 
departments and agencies of the Government of Canada at the mission. 


35 Inthe 2005-06 fiscal year, the Department spent $830 million 
on salaries and benefits out of a total budget of approximately 


$2.1 billion. 
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Human resources challenges 


3.6 Canada relies on the people who work in its missions to be its 
“face abroad.” The advent of globalization, pandemics, and terrorism 
affects the way the Department conducts its business and manages its 
employees. As a result, the Department encounters growing demands 
for its services from other government departments, clients such as 
Canadian companies needing export assistance, and individual 
Canadians. 


3.7 Employees work out of the Department’s headquarters in 
Ottawa, but staff are also posted to missions around the world. 
These postings involve varying degrees of hardship due to health and 
safety concerns and living conditions for the staff and their families. 
Since 1991 the number of missions rated with the highest level of 
hardship (considered dangerous or difficult) has risen from 15 to 32. 


3.8 In addition to managing staff around the world, another of the 
Department’s unique human resources challenges lies with the 
management of the following three distinct categories of employees 


(Exhibit 3.1). 


¢ Canada-based non-rotational staff are employees based in Canada 
who do not rotate locations. Most of these employees work at 
headquarters in Ottawa, and most who are non-executive are 
unionized. 


¢ Canada-based rotational staff are employees based in Canada who 
can be required to rotate every two to four years to a position at 
headquarters or at missions abroad. Most of this group who are 
non-executive are unionized. 


* Locally engaged staff (LES) are employees who work in Canada’s 
missions abroad who are mostly from the local workforce. They 
are not unionized and are subject to various local labour laws. 


3.9 Locally engaged staff are subject to local laws, but also to some 
Canadian laws and regulations since they work for the Government of 
Canada. Canada employs about 5,100 LES at its missions around the 
world. Roughly 1,500 of them work for programs led by Citizenship 
and Immigration Canada or by other government departments. 


3.10 Another human resources management challenge is presented 
by the Public Service Modernization Act, the provisions for which came 
fully into force in December 2005. The new legislation aims to simplify 
the competitive process and shorten the time it takes to fill positions. 
However, it requires significant adaptation on the part of both 
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employees and line managers in all departments to accommodate the 


new responsibilities and accountability. For example, managers can 


now take other factors into consideration when making decisions to 


fill positions. These include not only essential qualifications for the 


position, but also other qualifications considered assets for the work 


to be performed. Current and future operational requirements can 


be considered. 


3.11 


Finally, the Department also had to deal with organizational 


issues that arose from the government's decision to split the 


Department in 2003 into two organizations (the Department of 


Foreign Affairs and the Department of International Trade) and to 
subsequently reunify it in 2006. 


Exhibit 3.1 Department staff fall into three categories 


Number of employees 


2,800 


Canada-based non-rotational staff 


Canada-based rotational staff 


2,100 


Locally engaged staff (LES) 


5,100 


(as of 31 March 2006) 
Location of employment Canada Canada and abroad Local missions abroad 


Official languages 


English and/or French 


English and French both 
required for new recruits 


At least one official language 
usually required 


Foreign language requirement 


Not usually required 


Required for some postings 


Local language usually 
required 


Nationality and immigration 
status when hired 


Preference given to Canadians 


Preference given to Canadians 


Preference is given to 
Canadians, but most are 
foreign 


Acquisition of position 


Appointed to a position 


Chosen from a pool of eligible 
staff 


Hired to a position 


Requirement to relocate 


No expectation 


Expected to relocate every two 
to four years 


Not applicable 


Eligibility for financial 
compensation when residing 
abroad 


Eligible 


Eligible 


Not applicable 


Security clearance level 


Terms and conditions of 
employment 


Varies according to position 


Collective agreement (for non- 
executives); Public Service 
Employment Act (PSEA) 


Public Service Labour 
Relations Act 


Terms and Conditions of 
Employment for Executives 


High level necessary 


| Collective agreement (for non- 
| executives); Public Service 
Employment Act (PSEA) 


Public Service Labour 


Relations Act 


Terms and Conditions of 


| Employment for Executives 


| Minimal level required; 


higher levels difficult to obtain 


LES Employment Regulations 
| (Canada) 


Staff Engaged locally outside 
Canada (TBS policy) 


Subject to local laws of 


' country 
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Human resources planning 


Focus of the audit 


3.12 This audit looked at how well the Department carries out the 
planning of its human resources, and then manages and supports its 
workforce. We took a Department-wide approach to the audit rather 
than focusing solely on the Department’s Human Resources Branch. 
We focused on areas that we considered to be the most critical to 
achieving the Department’s objectives and that presented the 
highest risk. 


3.13. More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


3.14. Human resources planning is a process that identifies current 
and future human resources needs for an organization to achieve its 
goals. Human resources planning should serve as a link between 
human resources management and the organization’s overall 
strategic plan. 


3.15 Integrating human resources planning with business planning is 
also important to the Department of Foreign Affairs and International 
Trade’s understanding and assessment of current and future needs and 
challenges. The Public Service Employment Act and the Treasury Board 
Secretariat Management Accountability Framework point to the 
importance of integrated planning for departments and agencies. 


3.16 At the Department, decisions made about human resources also 
have ramifications on property management, relocation costs, 
information technology, and security and safety measures. Proper 
human resources planning can provide information about the level of 
risk involved in a variety of strategic management approaches and 
present senior management with viable alternatives to mitigate 
identified risks. 


3.17. While not unique to the Department, changing demographics 
also present human resources challenges. According to departmental 
information, 58 percent of its employees in the management category 
will be eligible for retirement by 2010; for all categories combined the 
figure is 26 percent. A recent study on human resources needs in the 
public service noted that “although these (public service) levels of 
retirement eligibility do not precisely indicate the volume of 
retirements to be expected in specific years, because most employees 
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do not leave immediately upon becoming eligible to retire, they do 
suggest a near-term acceleration of departures from the Public 
Service.” The most recent Public Service survey (2005) also showed 
that 37 percent of Department of Foreign Affairs and International 
Trade respondents said they would consider leaving the Department 
within the next five years. 


3.18 Departures now outnumber new hires (Exhibit 3.2). The number 
of hires is dropping, while the number of departures is increasing, 
making the need for strategic human resources planning, by using 
up-to-date workforce data, even more acute. If this trend continues, 
the situation will only worsen given the demographics already noted. 


Exhibit 3.2 Canada-based staff departures now outnumber new hires 


Number of Hires/Departures 


S 
350 
340 


330 
320 
310 
300 


290 
280 
270 


2004 2005 2006 


Source: Department of Foreign Affairs and International Trade 


The Department has no strategic human resources plan 


3.19 We expected the Department to have a strategic human 
resources plan that describes the complement and competencies of its 
current Canada-based and locally engaged staff, and those needed in 
the future. The plan should also outline departmental strategies to fill 
its workforce gaps in the short, medium, and long term and should be 
aligned with departmental objectives and priorities. 


3.20 We found that the Department does not have such a plan and, 
as a result, does not have a complete picture of the people, 
competencies, and experience it will need in the future. It lacks basic 
information that helps to plan for and manage its workforce. Until 
recently, human resources planning in the Department was limited. 
There was no link to business planning, and it was largely not 


integrated with other human resources management activities. 
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Previous Auditor General of Canada reports 
on government-wide recruitment 


2000 Report of the Auditor General of Canada, 
December, Chapter 21, Post-Secondary 
Recruitment Program of the Federal Public 
Service 


2003 Report of the Auditor General of Canada, 
2003 Status Report, Chapter 5, 
Post-Secondary Recruitment Program of the 
Federal Public Service 


Available human resources information was not effectively used for 
planning purposes and tended to focus on Canada-based staff only. 

As we note in later sections of this report, recruitment, promotion, and 
assignment processes were carried out reactively to fill immediate or 
urgent needs. 


3.21 The Human Resources Branch acknowledged that the limited 
attention paid to human resources planning activities in prior years 
had resulted in a number of problems, including vacancies in key 
occupational groups (executive, foreign services, and administrative 
services), a reactive instead of proactive response to needs, and 
overuse of temporary staffing measures (known as “acting” 
appointments). It also acknowledged the need for a greater focus on 
human resources planning due to the government's human resources 
management modernization initiative, among other issues. 


The Department is developing a strategic human resources plan 


3.22 We brought the lack of strategic human resources planning at 
the Department to Parliament’s attention in previous reports on 
government-wide recruitment. Under the Public Service Modernization 
Act, departments are expected to carry out strategic human resources 
planning. Under central agency guidance, the Department began to 
adopt the proposed human resources planning approach in 2005 and 
the Human Resources Branch was mandated to develop a human 
resources planning framework. The approved approach is extensive 
and guides the Department to determine its business goals, scan its 
environment, conduct a gap analysis, and set human resources 
management priorities to help achieve its business goals. Once the 
planning approach is implemented, its progress needs to be measured, 
monitored, and reported to senior management and central agencies. 


3.23 We also noted some good planning initiatives for specific groups 
coordinated by the Human Resources Branch. For example, in 

late 2005 a strategy was initiated to conduct strategic human resources 
planning for the Management and Consular Officer (MCO) pool. 
This strategy acknowledged, among other things, the importance 

of managers’ involvement in the success of its implementation. 

A standing committee was formed, which developed a strategy to 
address recruitment, development, and retention of staff in the MCO 
stream. Demographics, promotions, and assignments were analyzed 
and projected for MCO employees. The work also identified the 
critical need for the development of new recruits prior to the expected 
mass retirements. The committee developed an action plan with cost 
estimates, timelines, and deliverables. On the whole, this initiative 
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demonstrates the fundamental components required for sound human 
resources strategic planning. 


3.24 At the end of our examination, the Human Resources Branch 
was in the early stages of collecting data to prepare a strategic plan. 
However, it had not included the requirements for locally engaged staff 
in its planning process. In order to ensure that the Department has a 
sustainable workforce to fulfill its mandate, comprehensive human 
resources planning is fundamental to meet its demographic challenges 
and workforce gaps. 


The lack of workforce data is hampering human resources planning and 
management 


3.25 The preparation of strategic and operational plans requires 
reliable data on which to base analyses and projections. Our audit 
work raises questions about the reliability and usefulness of the 
Department's human resources management information data, such 
as acting levels and vacancies. Some managers tend not to use the 
departmental human resources information that is in the departmental 
systems, preferring to keep their own individual records. In addition, 
we found a number of errors in the data provided to us, and the 
aggregate numbers used in reporting to Parliament did not agree with 
the information in the departmental systems. 


3.26 Although the Department has been aware of a number of the 
problems with its human resources data for some time, it has not 
succeeded in correcting many of them. 


3.27. Recommendation. The Department should develop and 
implement a comprehensive human resources plan that clearly defines 
its current and future human resources needs, both for Canada-based 
staff and locally engaged employees. 


Department’s response. The Department agrees that a 
comprehensive human resources plan must be developed and 
implemented. In the last few years, significant progress has been made 
in establishing the capacity and systems necessary to plan in a complex 
and multi-dimensional environment. The development of a 
comprehensive Canada-based staff plan is in the final planning stages 
and is expected by spring 2007. Building on the Management Consular 
Officer model, the Department is developing plans for key professional 
groups in order to ensure that the Department has the right people in 


the right place doing the right job. 
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For locally engaged staff (LES), the Department will develop a 
framework by winter 2008 to guide missions with their LES planning, 
focusing on identifying key competencies required for its LES 
workforce. These mission-specific plans are based on local labour laws, 
and current and future human resources needs. There is little or no 
LES mobility between missions, the LES are not subject to the Public 
Service Employment Act (PSEA), and each country has its own set of 
terms and conditions of employment. As such, resourcing strategies 
and plans for the LES should be distinct from the Canada-based staff 
population, although intricately linked to the Department's current 
country strategy process. This process analyzes mission, policy, 
government, and other requirements and develops a mission-specific 
strategy ensuring that this is placed into an overall government 
context. These strategies then form part of the Department’s overall 
business plan. 


Workforce management 3.28 As we have noted, the Department of Foreign Affairs and 
International Trade faces particular human resources management 
challenges, especially with its demographics. This highlights the need 
for the Department to ensure that it is effective in filling its positions 
with enough qualified staff. 


3.29 We expected the Department to manage its human resources to 
ensure it has the appropriate number of employees with the required 
skills and competencies at the right time and in the right place. 

We also expected that the roles and responsibilities regarding 
recruitment, promotion, and job assignments would be clearly defined 
and well communicated. 


3.30 We examined three main activities—recruitment, promotion, 
and assignment of rotational staff—to determine whether the 
Department’s human resources policies and practices are current and 
consistent with good human resources management practices, and if 
they support departmental objectives. In addition, we looked at the 
management of locally engaged staff working in missions abroad, 

and the process for compensating Canada-based staff posted to 

these missions. 


3.31 The following sections present our findings in a variety of areas. 
An overall recommendation is provided at paragraph 3.64. 
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Relying on traditional recruitment methods might not be sustainable 


3.32 We expected the Department to have a clear picture of the 
vacancies it is trying to fill and to have developed and implemented 
practices to ensure that sufficient candidates with the required skills 
are selected to meet workforce needs. 


3.33 The Department’s human resources information system showed 
a 35 percent vacancy rate as of 31 March 2006. During the course of 
our audit, the Department informed us that according to its best 
estimate, its vacancy rate was approximately 20 percent, although 
about half of those positions were inactive—that is, they were 
considered redundant but had not been taken off the organizational 
charts. We could not validate those figures and the Department could 
not provide an exact number of vacancies for which it was trying to 
recruit. The impact of this estimated vacancy rate is that there are a 
good number of positions that do not have an incumbent to deliver 
services and programs to Canadians both in Canada and abroad. 

We found no clear link between recruitment levels and long-range 
planning, demographic analysis, or vacancy-trend analysis. 


3.34 In our 2000 chapter on post-secondary recruitment, we noted 
that the Department did not link its recruitment with attrition rates 
and retirement data. For the 2005-06 recruitment process, we found 
that the Department produced some analysis of its needs for foreign 
service officer recruitment, but was unable to show a link between its 
needs and the actual number of hires. 


3.35 Historically, the Department has filled its rotational positions by 
recruiting candidates at the entry level and subsequently promoting 
them throughout their careers. It is extremely rare for non-rotational 
Department and other public service employees to enter the rotational 
stream other than through the competitive entry-level process. 


3.36 In order to meet immediate operational requirements for 
experienced, highly skilled officers at the more senior levels of the 
foreign service officer group, the Department launched a staffing 
exercise in December 2005. This process would have provided non- 
foreign service employees, both from within the Department and from 
other departments, with the opportunity to transfer into the foreign 
service group at a classification higher than entry level. This is known 
as lateral entry. This type of entry represented a radical shift from the 
traditional method of recruitment that restricted these positions to 
foreign service officers, and was contested before the courts by the 


foreign service officers’ union. 
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3.37 The Department’s decision to conduct this exercise was set aside 
by the Federal Court in September 2006 on the basis that the 
Department had not developed its own deployment policy, as required 
under Treasury Board policy, and had not consulted the Professional 
Association of Foreign Service Officers. The Department believes it 
has since satisfied the court’s conditions and has redeployed the 
affected employees under provisions of the new Public Service 
Employment Act. 


3.38 We also examined a representative number of entry-level 
competition files for the rotational foreign service and administrative 
services categories to assess the recruitment process in the 2005-06 
fiscal year. The competitions were generic, allowing for one staffing 
process to fill similar positions within the Department. We reviewed 
the sources of recruitment, the qualification steps, the selection tools, 
and the documentation used for the staffing process. 


3.39 Overall, our sample of entry-level competition files indicated 
that the recruitment process was properly established and followed in 
the selection of candidates, although more rigour is needed in the 
interview process and reference checks for the administrative 
rotational staff. Although management roles and responsibilities were 
defined and well communicated, we noted instances where they were 
not carried out appropriately. 


The Department cannot fill its needs on a timely basis through promotions 


3.40 The Department holds promotion competitions for both 
rotational and non-rotational Canada-based staff. While non- 
rotational employees are promoted to a specific position, rotational 
employees are promoted to a level with no predetermined job 
assignment. Promotions are especially important to both the 
Department and its rotational employees because, as previously 
mentioned, the Department has traditionally filled rotational positions 
above entry-level almost exclusively through promotions. Promotion 
processes intended to fill rotational assignments take the form of 
generic competitions. 


3.41 We expected that the Department would have a suitable 
promotion process that would contribute to filling the workforce gaps 
with qualified candidates on a timely basis. 


3.42 We examined a representative number of individual promotion 
files for both rotational and non-rotational Canada-based staff for 
the 2005-06 fiscal year. We examined whether appropriate tools 
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were used throughout the processes to assess the qualifications of 
candidates, such as position descriptions, competency profiles, 
interview guides, and reference checks. We found that the Department 
did, in fact, use appropriate tools, consistent with good human 
resource management practices. As well, we found that management 
roles and responsibilities were properly carried out. 


3.43 We also found that the competitive process took from 1 to 

22 months depending on the type of competition (single actions to fill 
a single position, or generic staffing actions for several positions). In 
one third of the files examined, the process was generic and took over 
six months. 


3.44 Ina 2006 government-wide study, the Public Service 
Commission estimated that the average time for promotion in the 
Public Service is 23 weeks. The study noted that government 
departments that run larger (generic) processes take longer to staff 
positions. We found that at the Department, such competitions require 
the assessment of a large number of candidates who may already be 
posted around the world. Other factors, such as language testing and 
security clearance, may also add to the length of a process. 


3.45 We also found that the Department is having difficulty filling 
some of its workforce gaps through promotions. For example, in 
April 2005, the Department launched a promotion competition for 
executives at the EX-01 classification level. Although it had not set a 
specific number of positions to be filled by this process, the 
Department hoped that the competition would allow it to promote as 
many successful candidates as possible. 


3.46 During this process, applicants were pre-screened, and 370 were 
found to meet the experience requirements. We noted that 132 passed 
the first qualifying step; however, in May 2006, only 39 successful 
candidates were promoted. Some unsuccessful candidates appealed the 
decision and at the end of our audit, there had been no final decision 
on the appeals. There has been no other promotion process launched 


for EX-O1 positions since then. 


3.47 As of 31 March 2006, 116 non-executive category employees 
were acting in executive positions. We found that 35 of those had not 
demonstrated the competencies to pass the first step of the ongoing 
process for promotion to the EX-01 level. These results demonstrate 
that the Department might not be able to rely predominantly on 


promotions to fill its workforce gaps at the executive level. 
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3.48 Several managers and employees told us that they found the 
various promotion processes frustrating and ineffective. Some of the 
factors mentioned were the length of the processes, the negative 
impact on morale, and the inability of candidates already in acting 
positions to pass the qualifying steps to be made permanent in that 
position and at that level. 


3.49 In 2006 many collective staffing actions were completed, a 
number of staffing processes were under way at the time of our audit, 
and several others were slated for completion by the spring of 2007. 
These staffing actions are due in part to the fact that the Human 
Resources Branch confirmed a number of shortages in key areas, in 
particular within the foreign service and administrative services 
streams, but also among several other groups. 


3.50 Given its demographic trends that predict increased departures, 
it will be important for the Department to ensure, as part of its human 
resources strategic planning exercise, that its recruitment and 
promotion strategies result in having the human resources it needs to 
carry out its mandate. 


The assignment process does not yield the intended results 


3.51 All rotational staff, including executives, foreign service officers, 
and administrative personnel, are placed in a pool and assigned to 
positions abroad or at headquarters for terms of two to four years. 

It is important that staff meet the requirements of the positions they 
are appointed to. For employees, the assignment process determines 
where they and their families will live for a number of years and has 
consequences for their career paths. 


3.52 Available assignments abroad are posted once a year. Assignment 
decisions are to be made based on a list of key elements, including 
departmental need and employee preference, among others. 


3.53 The section at headquarters responsible for a given mission 
participates in the initial pre-screening of all applicants and interviews 
the candidates for a particular posting. After various consultations 
throughout the process, a final candidate is selected by both the 
corporate assignment board (a committee of senior Department 
managers who review and approve assignments) and the head of 
mission. 


3.54 At the Department, employees from the non-EX groups in acting 
positions receive acting pay, while those from the rotational EX group 
do not—which is the deputy minister’s prerogative. Given the 


14 


Chapter 3 


Report of the Auditor General of Canada—May 2007 


HUMAN RESOURCES MANAGEMENT—FOREIGN AFFAIRS AND INTERNATIONAL TRADE CANADA 


rotational environment, employees assigned to positions above their 
job classification levels are generally assigned for the number of years 
of the rotation, not; on a short-term basis. 


3.55 We expected that the assignment process would achieve the 
Department's intended results of placing the right people at the right 
place at the right time. We examined whether the assignment of people 
to rotational positions matched skills to requirements. 


3.56 We observed a practice of assigning staff to rotational positions 
regardless of whether their level matched the job requirements, and we 
found a significant number of people in acting positions for periods 
such as a whole posting cycle. 


3.57. We noted that as of 31 March 2006 for the whole Department, 
both at headquarters and in the missions, there were 917 assignments 
of Canada-based staff (128 executive and 789 non-executive) at 

a position level that did not match the employee’s classification level. 
The positions of the FS, AS, and EX groups each had around 

30 percent of acting appointments. 


3.58 As of 31 March 2006, of the approximately 900 Canada-based 
staff in rotational positions abroad, more than 300 were at a level that 
did not match the employee’s classification level. About 100 of those 
were in assignments at least two levels above their own. The fact that 
there are so many Department employees in acting positions, and often 
for periods of up to four years, could raise questions about whether 
employees are fully qualified to fill their acting roles. 


3.59 We reviewed a representative sample of “acting” assignment files 
as of 31 March 2006, together with other assignment files. We found 
little or no documentation as to why the person was selected or to 
what extent the individual selected met the requirements of the 
position. We reviewed minutes of the selection board meetings and 
found little rationale supporting the assignment decision. In some 
cases, we found a statement indicating that the person was well 
qualified, but there was nothing on file to support this. 


Foreign language requirements are not met 


3.60 In countries where French or English is not the local language, 
foreign language skills are key to adequately representing Canada’s 
interests. 

3.61 We expected that the Department would have identified those 
positions that required foreign language proficiency and taken steps to 


ensure that incumbents met the requirements. 
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3.62 Recently, the Department conducted an analysis of its foreign 
language skills requirements for each of its missions and identified 
180 positions that required a general working proficiency in a foreign 
language. 


3.63 We found that the Department has not been able to recruit, 
train, and assign enough rotational staff in order to fulfill these 
language skills requirements. Only 16 percent of the people occupying 
these positions met the language requirement of their position and 

33 percent of incumbents.had not undergone language testing. 


3.64 Recommendation. As part of its human resources plan, the 
Department should outline its strategies to fill current and projected 
gaps in its workforce, including how recruitment, promotion, 
assignment, and language training activities can best contribute to 
ensuring that it has qualified people in the right place at the right time. 


Department’s response. The Department agrees with the 
recommendation and recognizes that strategies to fill current and 
projected gaps in its workforce are important elements in human 
resources planning. Recent changes to the departmental configuration, 
as well as the conversion in 2005 of the Foreign Service group, have 
had a significant impact on data integrity. Data integrity and the 
management of vacant positions are key success factors in determining 
immediate and longer term needs. To address this, the Department has 
already created a unit headed by an executive to focus on human 
resources data integrity and anticipates that it will take up to two years 
to update all the systems to ensure quality data. 


The Department is working with the Statistics Canada Centre for 
Workforce Analysis and Forecasting to modify its systems and improve 
its ability to address workforce gaps. 


As suggested in this report, the Department agrees and will continue 
to use non-traditional means to staff positions, whenever appropriate. 
The Department agrees that foreign language training needs must be 
better linked to the staffing conducted by the Department and will 
endeavour, by winter 2008, to study foreign language training 
requirements and develop an action plan. 


The management of locally engaged staff gets little attention 


3.65 Under Treasury Board policy, the Department uses locally 
engaged staff when it is practical and when it is in the public interest. 
Due in large part to successive budget pressures over the last two 
decades, the proportion of staff working in Canada’s missions who are 
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locally engaged is on the rise. As a result, among Western democracies, 
Canada has one of the highest ratios of locally engaged staff in its 
foreign missions. We expected that under these conditions, the 
Department would have complete and accurate human resources 
information and strong management support to ensure that its local 
workforce was optimally employed, and that managers who rotate 
every two to four years would be able to manage this workforce 
consistently. 


3.66 We found that the Department does not have accurate and 
complete information to properly manage these employees. We noted 
instances of staff recorded as working at the wrong mission, of missions 
without complete employment histories of their staff on file, or of 
training that was not fully tracked. In addition, we found in some cases 
no record of an oath of office taken by local staff, as required by the 
LES Employment Regulations, or no record of their security checks. 


3.67. The Locally Engaged Staff Services Bureau at headquarters in 
Ottawa is responsible for developing and implementing the 
departmental policy for the management of locally engaged staff 
abroad, and for providing services and support to all Canadian 
missions. At missions abroad, management consular officers are 
responsible for consular and all administrative functions, including the 
support to the management of human resources. The day-to-day 
management of these employees is carried out by mission managers 
who are generally rotational Canada-based staff. 


3.68 We looked at how the LES Services Bureau and missions abroad 
carry out their responsibilities in managing the locally engaged staff. 
We expected that Department headquarters would provide appropriate 
support to mission managers. The Bureau’s support and monitoring 
activities are particularly important because, in most missions, the 
manager overseeing the human resources management function is 
rotational and not necessarily a human resources professional aware of 
all the complexities of managing staff who are subject to local laws and 


customs. 


3.69 We found that the Bureau had difficulty in carrying out all 
its responsibilities and has been unable in recent years to fully meet 


its mandate. 


3.70 For instance, the Bureau is struggling to keep the LES handbooks 
up to date for each of Canada’s missions. The LES Handbook 
developed for each mission is an essential document that provides 


specific terms and conditions of employment that are part of Canada’s 
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contract with local staff. The handbooks not only provide key material 
concerning pay and benefits for the staff but also include crucial 

local information for newly arrived Canada-based managers. The 
Department considers that each handbook needs to be updated 

at least every four years. At the current rate of progress, missions can 
expect their handbook to be updated about once every 15 years. 


3.71 As part of its mandate, the Bureau attempts to visit missions to 
conduct on-site reviews of their human resources management practices 
and review issues such as overpayments of salaries or benefits, errors in 
the classification process, or poor record keeping. We noted that in 
recent years, the Bureau visited only four missions per year on average. 


3.72 Recommendation. The Department should strengthen the 
management of its locally engaged employees and better support 
mission managers in carrying out their responsibilities. 


Department’s response. The Department agrees that the capacity 
to deliver quality services to the locally engaged staff (LES) 
population and their management teams needs to be strengthened. 
In the 2007-08 fiscal year and beyond, the Department will 
undertake to re-examine the current business model. As part of this 
re-examination, the type of support that is required and what form 
it should take will also be examined, and the experience of other 
governments will be sought in order to obtain best practices used 
around the world. 


By winter 2008, the Department will develop a strategy to review 
benefits provided to LES employees around the world, including the 
LES Employment Regulations, terms and conditions, classification, 
and human resources delegation authorities. 


There is little flexibility to compensate and provide incentives to staff for 
living abroad 


3.73 As previously noted, the Department is facing a rapidly changing 
environment that requires it to respond to the needs of Canadians in a 
timely manner. Its rotational staff are on the front lines of its program 
delivery and their personal situations are also affected by this rate of 
change. Many missions are in environments that compare poorly with 
Canada in their standard of living, security, safety, and health care. 
Missions abroad are rated on a hardship scale of 0 to 5. As a result of 
increasing difficulty with security and related issues, the number of 
Level 5 missions (considered dangerous or extremely difficult) has 
risen from 15 to 32 since 1991. 
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3.74. We examined whether the Department has processes in place to 
allow it to respond in a timely manner to the changing circumstances its 
employees are facing. We expected the Department and the employer 
(the Treasury Board) to have flexible mechanisms to enable the 
Department to assign qualified staff to its missions around the world. 


3.75 We found that since the 1960s the Department has principally 
relied on Foreign Service Directives (FSDs) as an important tool to 
serve this purpose. The Directives address a number of issues that 
government departments must consider when assigning staff to 
missions, such as differences in living conditions, cost of living, travel 
home for holidays, children’s schooling or day care, adequate housing, 
medical care, and safety and security. These issues also influence the 
Department's ability to assign or retain employees and deliver 
programs effectively at its missions. 


3.76 The FSDs are deemed to form part of the collective agreements 
and also apply to departmental executives posted abroad. Any changes 
to the FSDs must be co-developed by a sub-committee of the National 
Joint Council (NJC), a consultative body of 18 bargaining agents, 

the Treasury Board Secretariat as the employer’s representative, and 
three separate federal agencies. They are normally reviewed every 
three years. The Department can influence the process through 
submissions to the employer and through its participation in NJC 
working groups. 


3.77 The effectiveness of the Foreign Service Directives processes is 
a long-standing issue. As far back as 1987, our Office commented on 
the complexity of the Directives and questioned the procedures for 
determining whether they were achieving their stated intent. 

We reiterated our concerns in our 1994 and 1997 reports, and 
suggested that effecting change would require that the government 
fundamentally re-examine the FSD system and how it is managed. 
Since then, a 1995 special steering committee, the National Joint 
Council, the Department’s employees, and the Department itself have 
all expressed concern about the processes for developing and applying 
the FSDs. However, the processes have remained essentially the same 


to this day. 


3.78 The limitations of the FSD processes can affect the ability to 
deliver longer-term strategic initiatives. For example, the Government 
of Canada launched the high-priority Enhanced Representation 
Initiative (ERI) to fulfill its commitment to establishing stronger 
representation across the United States (U.S.). Through the ERI, 
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Canada now has 22 consulates and consulates-general throughout the 
US, in addition to the embassy. 


3.79 The Department and other ERI partner departments are having 
difficulty convincing their staff to go to these missions in the US, and 
when they do go, some are unwilling to stay. In 2006 there were only 
3.8 applicants for each US posting, compared with 9.3 for similar 
postings elsewhere and 6.6 overall. In recent years, 73 percent of 
employees who voluntarily terminated their postings early at their 
own expense were at US missions, which account for only about 

15 percent of all Canada-based staff abroad. The allowances provided 
under the directives for living in the US are seen by rotational staff and 
their families as a disincentive to accepting a US posting (Exhibit 3.3). 


Exhibit 3.3 Foreign Service Directives for US missions are seen as a disincentive to staff assignments 


US missions — 

Housing Employees are usually required to Generally provided in fully furnished 
arrange for their own leased housing and Crown-owned or Crown-leased housing. 
its maintenance. FSDs provide a rental The Crown provides maintenance and 
allowance but do not cover all related furnishings. 
costs. 

Education for children Schooling is provided by local schools, Schooling is paid for by FSDs and 
as mandated by FSDs. Quality of public provided at private schools also attended 
schooling depends on the local school by children of the diplomatic community. 
district, which depends on available 
housing. 

Cost of living According to the Department, costs are According to the Department, the “post 
18 percent higher than in Canada, but living allowance” FSD adequately 
employees in the US are not fully compensates for the higher cost of living. 
compensated under FSDs to this amount. 


3.80 In March 2005 the Department requested changes to the FSDs 
to improve compensation and incentives to employees posted to the 
US. In its oversight capacity, the Treasury Board Secretariat requested 
a business case with evidence to substantiate the request. The 
Secretariat found that the business case did not meet its criteria. At 
the time of our audit, the Department was working with the Treasury 
Board Secretariat on a revised business case. 


3.81 The Department recognizes that the FSD system does not 
provide it with adequate flexibility to respond to changing 
circumstances on a timely basis. The Department has recently 
explored the possibility of working with the Treasury Board Secretariat 
as the employer’s representative to find alternative solutions to urgent 
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issues. One such issue is that few rotational employees want to go on 
high-risk postings, such as Canada’s mission in Afghanistan. 

In July 2005 the Department began making a case to persuade the 
Treasury Board Secretariat to help put in place special measures prior 
to the next cyclical review of the Foreign Service Directives. Some 
of the measures the Department eventually proposed in June 2006 
included the following: 


* The deputy minister be delegated authority to provide life 
insurance for employees serving in Afghanistan. Frequently, 
private life insurance of employees is invalid in a war zone. 
Consultations with employees suggested that the lack of adequate 
insurance against hostile acts is one of the most serious barriers to 
accepting high-risk assignments. 


* The hostility bonus for the most difficult postings, such as 
Afghanistan be raised from 50 percent of the hardship allowance 
to 250 percent of the hardship allowance. 


¢ Entitlement to full FSD compensation be given to employees who 
spend at least four months at a high-risk mission. Full FSD 
compensation is normally limited to employees who spend a 
minimum of one year at a mission. 


¢ A tax exemption on their earnings be given to employees while 
on dangerous missions, which is consistent with the tax 
exemptions implemented in 2004 for the Canadian Forces. 


3.82 At the time of our audit, the Treasury Board Secretariat had just 
clarified that the deputy minister could use his delegated authority 
under the Financial Administration Act to purchase adequate life 
insurance policies. The other measures were still under discussion. 
While this effort shows that it is possible to find alternative solutions 
outside the triennial Foreign Service Directive review process, other 
outstanding issues have been in discussion for more than a year. 


Barriers to spousal employment are disincentives for working abroad 


3.83 Another issue of particular and long-standing concern is spousal 
employment. As far back as 1981, the Royal Commission on 
Conditions of Foreign Service reported that the effect of the foreign 
service lifestyle on employees and families was the second most 
common reason for resignations. The demanding nature of the posting 
cycles, family stresses, and the lack of employment opportunities for 


spouses were reasons clearly influencing decisions to resign. 


— 
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3.84 With the increase in two-income families, the issue of spousal 
employment has only become more acute. Staff members posted 
abroad often face the difficult choice of separation from their spouse or 
the loss of one family income, pension benefits, and the postponement 
or loss of the spouse’s career. While there are additional amounts 
included in the FSD allowances when a staff member is accompanied 
abroad, they are not designed to address the spousal employment issue. 
Spousal employment abroad may be difficult to obtain due to a number 
of barriers, such as language difficulties at each new post, local 
restrictions on work permits, and lack of recognition of professional 
qualifications. Following the loss of Canadian employment on 
departure, spouses are not eligible for employment insurance while 
looking for a job in the foreign country (except in the United States). 
The impact of these barriers can be significant for the family posted 
abroad, as well as for programs and service delivery at missions. 


3.85 Departmental senior management recognizes this issue as a 
serious one. The Department has put in place some support programs 
and activities. It has surveyed what other countries do for their families 
posted abroad who face the same types of problems. Some of those 
countries have explored alternative solutions, such as a significant 
spousal allowance or reserved positions at missions for spouses. 


3.86 It is important to note that the issues raised in this and the 
previous section of the audit may be shared by a number of other 
government departments and agencies who assign staff abroad. 


3.87 Recommendation. The Treasury Board Secretariat, with the 
active participation of the Department of Foreign Affairs and 
International Trade and other departments and agencies, should 


¢ lead a comprehensive review of the challenges facing the Public 
Service in posting employees abroad; and 


* ensure that appropriate mechanisms are in place, either within or 
outside the Foreign Service Directive (FSD) framework, to allow 
departments and agencies to respond in a timely and effective 
manner to changing circumstances affecting the assignment of 
qualified staff around the world. 


Treasury Board Secretariat’s response. The Treasury Board 
Secretariat agrees. The Secretariat is committed to establishing and 
leading a partnership of key departments and agencies to 
comprehensively identify the challenges facing the Public Service 
in assigning staff abroad, and to developing and implementing a 
reasonable and effective plan of action in a time frame appropriate 
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to the challenges. The Secretariat will establish this partnership and 
commence this work by summer 2007. 


Department’s response. The Department agrees that a serious 
examination of the issues facing those being posted abroad is essential. 
Currently, most “FSD-related” issues are considered, but few 

of them are resolved through the National Joint Council (NJC). This 
co-development process does not easily permit priority focus on one 
issue, but sets all issues among the total FSD package. This negotiating 
process has proven to be quite cumbersome and the current FSDs have 
remained essentially unchanged for over 40 years, despite dramatic 
changes in Canadian society and changing government priorities. 


This framework seriously limits the Department of Foreign Affairs and 
International Trade’s and other departments’ ability to respond to new 
or evolving government priorities and programs, all the while being 
charged with the responsibility and accountability for the program 
delivery. In addition, a large number of employees unrepresented by 
any bargaining agent in the process find themselves subject to these 
regulations, although they have no appeal rights within the NJC 
forum. 


Attempts to support Canadian government employees serving in 
extreme high risk operations such as Afghanistan and Haiti are still 
being negotiated as part of the FSD process, which began in 2005. 


The Department welcomes the authority and flexibility to meet our 
obligations through a more streamlined and responsive FSD or other 
mechanism, or by obtaining separate authorities for operational needs. 
The Department suggests that the governance process should be 
re-examined. 


Conclusion 


3.88 The Department of Foreign Affairs and International Trade does 
not have a comprehensive human resources plan and does not have 
the information it needs to identify the staffing levels, competencies, 
and experience it requires now and in the future. 


3.89 Asa result of a lack of human resources planning, management 
of its workforce suffers from many deficiencies that put the attainment 
of its objectives at risk. Recruitment and promotion processes do not 
ensure that the position requirements are met on a timely basis in 
terms of numbers of vacancies filled or skills capacity. Assignments 

to rotational positions result in a high number of people whose 
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classification levels do not match the job requirements. The 
Department is not paying enough attention to the management of 
locally engaged staff. There is little flexibility to compensate and 
provide incentives to staff for the cost of living and hardship while 
working in foreign missions. 


3.90 The Department has begun to develop a planning process for its 
human resources that can support its long-term management needs. It 
has a number of good practices to build on, but a sustained effort is 
required to bring modern human resources planning and management 
into its culture. Proper human resources planning would contribute to 
closing the staffing gaps and responding to the Department’s 
immediate and longer-term needs. 
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About the Audit 


Objectives 


This audit assessed whether the Department of Foreign Affairs and International Trade adequately plans 
its human resources and manages its workforce with respect to the recruitment, staffing, and assignment of 
Canada-based and locally engaged staff in order to meet departmental objectives. 


Scope and approach 


Our audit focused on human resources planning, key areas of the Department’s workforce management 
(recruitment, assignment, and promotion processes, and the locally engaged staff human resources 
function at headquarters), as well as on the Department’s efforts to modernize its human resources 
management. 


We took a Department-wide approach to the audit rather than focusing solely on the Department’s 
Human Resources Branch. In this audit’s scope, we did not include the human resources management of 
Passport Canada employees and of the Canadian Foreign Service Institute, or of Canada-based staff 
working at missions for other government departments. Our audit also did not cover the human resources 
management functions of classification, training, career management, employee evaluation, or health and 
safety issues. We analyzed internal documentation and processes. File reviews were carried out at 
headquarters and at missions. Random sampling was done according to Office standards. Files reviewed at 
headquarters covered recruitment and promotion for several occupational groups and various elements 
(statement of qualifications, job description, rating guide, basic requirements, selection board composition, 
and so on) of the processes. We also reviewed files on acting appointments, for which a non-proportional 
stratified random sample was used. The population was stratified by category of classification, each of the 
classifications given equal treatment. We also quantified the demographic and competency shortfalls at 
the Department with data snapshots. 


At missions, contract files as well as Canada-based staff and locally engaged staff personal and position files 
were reviewed. We also compared employees’ foreign language competencies with position requirements. 


We conducted interviews with management and employees at headquarters and in 12 missions. 


Criteria 


We expected 

- human resources plans to be timely, complete, integrated with the Department's business plan, and 
responsive to the needs of the Department; 

* appropriate data for human resources planning and its management to be accurate and easily 
accessible; 

* the Department to be able to have the appropriate number of employees with the required skills and 
competencies at the right time and in the right place; and 

- roles and responsibilities for recruitment and job assignments to be clear, defined, and well 
communicated. 
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Audit work completed 


Audit work for this chapter was substantially completed on 28 November 2006. 


Audit team 


Assistant Auditors General: Ronnie Campbell and Richard Flageole 
Principals: Marie Bergeron and Paul Morse 
Director: Claude Brunette 


Rob Anderson 
Sylvie Deschamps 
Catherine Martin 
Heather Miller 
Robyn Roy 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 3. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Human resources planning 


3.27 The Department should develop The Department agrees that a comprehensive human resources 


and implement a comprehensive plan must be developed and implemented. In the last few years, 
human resources plan that clearly significant progress has been made in establishing the capacity 
defines its current and future human and systems necessary to plan in a complex and multi- 
resources needs, both for Canada-based —_ dimensional environment. The development of a comprehensive 
staff and locally engaged employees. Canada-based staff plan is in the final planning stages and is 
(3.14-3.26) expected by spring 2007. Building on the Management Consular 


Officer model, the Department is developing plans for key 
professional groups in order to ensure that the Department has 
the right people in the right place doing the right job. 


For locally engaged staff (LES), the Department will develop a 
framework by winter 2008 to guide missions with their LES 
planning, focusing on identifying key competencies required for 
its LES workforce. These mission-specific plans are based on 
local labour laws, and current and future human resources 
needs. There is little or no LES mobility between missions, the 
LES are not subject to the Public Service Employment Act 
(PSEA), and each country has its own set of terms and 
conditions of employment. As such, resourcing strategies and 
plans for the LES should be distinct from the Canada-based staff 
population, although intricately linked to the Department's 
current country strategy process. This process analyzes mission, 
policy, government, and other requirements and develops a 
mission- specific strategy ensuring that this is placed into an 
overall government context. These strategies then form part of 
the Department’s overall business plan. 
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Workforce management 


3.64 As part of its human resources 
plan, the Department should outline its 
strategies to fill current and projected 
gaps in its workforce, including how 
recruitment, promotion, assignment, 
and language training activities can best 
contribute to ensuring that it has 
qualified people in the right place at the 
right time. (3.28-3.63) 


3.72 The Department should 
strengthen the management of its 
locally engaged employees and better 


support mission managers in carrying 
out their responsibilities. (3.65—3.71) 


The Department agrees with the recommendation and 
recognizes that strategies to fill current and projected gaps in its 
workforce are important elements in human resources planning. 
Recent changes to the departmental configuration, as well as the 
conversion in 2005 of the Foreign Service group, have had a 
significant impact on data integrity. Data integrity and the 
management of vacant positions are key success factors in 
determining immediate and longer term needs. To address this, 
the Department has already created a unit headed by an 
executive to focus on human resources data integrity and 
anticipates that it will take up to two years to update all the 
systems to ensure quality data. 


The Department is working with the Statistics Canada Centre 
for Workforce Planning and Forecasting to modify its systems 
and improve its ability to address workforce gaps. 


As suggested in this report, the Department agrees and will 
continue to use non-traditional means to staff positions, 
whenever appropriate. The Department agrees that foreign 
language training needs must be better linked to the staffing 
conducted by the Department and will endeavour, by winter 
2008, to study foreign language training requirements and 
develop an action plan. 


The Department agrees that the capacity to deliver quality 
services to the locally engaged staff (LES) population and their 
management teams needs to be strengthened. In the 2007-08 
fiscal year and beyond, the Department will undertake to re- 
examine the current business model. As part of this re- 
examination, the type of support that is required and what form 
it should take will also be examined, and the experience of other 
governments will be sought in order to obtain best practices used 
around the world. 


By winter 2008, the Department will develop a strategy to review 
benefits provided to LES employees around the world, including 
the LES Employment Regulations, terms and conditions, 
classification, and human resources delegation authorities. 
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3.87 The Treasury Board Secretariat, Treasury Board Secretariat’s response. The Treasury Board 


with the active participation of the Secretariat agrees. The Secretariat is committed to establishing 
Department of Foreign Affairs and and leading a partnership of key departments and agencies to 
International Trade and other comprehensively identify the challenges facing the Public 
departments and agencies, should Service in assigning staff abroad, and to developing and 


implementing a reasonable and effective plan of action in a 
time frame appropriate to the challenges. The Secretariat will 
establish this partnership and commence this work by 
summer 2007. 


* lead a comprehensive review of the 
challenges facing the Public Service 
in posting employees abroad; and 


* ensure that appropriate mechanisms 
are in place, either within or outside 
the Foreign Service Directive (FSD) 
framework, to allow departments and 
agencies to respond in a timely and 
effective manner to changing 
circumstances affecting the 
assignment of qualified staff around 


the world. (3.73-3.86) 


The Department’s response. The Department agrees that a 
serious examination of the issues facing those being posted 
abroad is essential. Currently, most “FSD-related” issues are 
considered, but few of them are resolved through the National 
Joint Council (NJC). This co-development process does not 
easily permit priority focus on one issue, but sets all issues among 
the total FSD package. This negotiating process has proven to be 
quite cumbersome and the current FSDs have remained 
essentially unchanged for over 40 years, despite dramatic 
changes in Canadian society and changing government 
priorities. 


This framework seriously limits the Department of Foreign 
Affairs and International Trade’s and other departments’ ability 
to respond to new or evolving government priorities and 
programs, all the while being charged with the responsibility and 
accountability for the program delivery. In addition, a large 
number of employees unrepresented by any bargaining agent in 
the process find themselves subject to these regulations, 
although they have no appeal rights within the NJC forum. 


Attempts to support Canadian government employees serving in 
extreme high risk operations such as Afghanistan and Haiti are 
still being negotiated is part of the FSD process, which began 

in 2005. 


The Department welcomes the authority and flexibility to meet 
our obligations through a more streamlined and responsive FSD 
or other mechanism, or by obtaining separate authorities for 
operational needs. The Department suggests that the 
governance process should be re-examined. 
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Main Points 


The Canadian Agricultural Income Stabilization (CAIS) program is a 
federal-provincial-territorial program whose objective is to protect 
agricultural producers from drops in income due to circumstances 
beyond their control. The program is administered by Agriculture and 
Agri-Food Canada for all of Canada except Alberta, Ontario, Quebec, 
and Prince Edward Island, where the provincial governments 
administer the program. We did not audit the provinces’ delivery of the 
CAIS program. 


We conducted this audit at the request of the House of Commons 
Standing Committee on Agriculture and Agri-Food. We examined 
how the Department processes applications for income support, 
ensures that all parties are respecting the various monitoring provisions 
set out in the federal-provincial-territorial agreements, and measures 
and reports its performance to Parliament. 


The federal government has provided income support programs to 
farmers for decades. Agriculture and Agri-Food Canada receives about 
55,000 applications to CAIS each year. In the 2005-06 fiscal year, the 
Department spent about $1.1 billion Canada-wide, making the CAIS 
program the Department’s principal ongoing program of farm income 
support. 


- The CAIS program is very complex, and the Department does not 
make it transparent enough to producers; some have complained 
that they do not understand how the Department calculates their 
benefits. For example, when the Department changes financial or 
other information that producers submit, it does not explain the 
impact on the benefit amount. Consequently, producers do not 
receive, in an easy to understand manner, all the information they 
need to decide whether to ask for a payment adjustment or appeal an 
unfavourable decision. Further, there are long delays before 
producers are told whether they will receive a benefit and in what 


amount. 
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» The Department focuses on preventing and detecting overpayments 
made to producers, and the onus is on producers to question 
incorrect figures in the Calculation of Program Benefits statement 
sent by the Department. Improving transparency would make 
program delivery more consistent with the program’s objective, 
as set out in the Farm Income Protection Act—to protect the income 
of producers. 


Payment errors have been high, although they are beginning to 
decline. Staff who calculate payments work to standards that 
emphasize the number of applications processed and adherence to 
procedures more than accuracy of payments. While the Department 
collects data on the amounts and nature of the errors, it does not 
systematically use that information to improve its management of 
the program. 


- Some employees who processed applications also worked as private 
consultants to help producers prepare their applications. We are 
concerned that such conflicts of interest could provide an unfair 
financial advantage to some applicants. 


» Some of these problems are not new; our past audits of previous farm 
income support programs have found similar problems. 


The Department has responded. In its response to each 
recommendation throughout the chapter, Agriculture and Agri-Food 
Canada has indicated the actions it has taken, will take, or will 
consider taking. 
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Agricultural producers—Includes those who 
produce 


° 


livestock and poultry and related products 
(for example, eggs, milk, meat, and animal 
fibres) and ranch fur products (for example, 
mink, fox, chinchilla, and rabbit): 


crop and plant products (for example, wheat, 
barley, oats, canola, corn, fruit, nuts, 
mushrooms, wild rice, maple syrup, and 
herbs); and 


nursery products (for example, trees, shrubs, 
annual and perennial plants, cut flowers, and 
cultivated Christmas trees). 
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Introduction 


Agriculture and Agri-Food Canada’s administration of farm income support 
programs 


41 Agriculture and Agri-Food Canada has administered various 
farm income support programs since the late 1930s. The Canadian 
Agricultural Income Stabilization (CAIS) program is the latest in a 
number of programs that Agriculture and Agri-Food Canada 
developed to provide income support to agricultural producers. 


42 In 1991, the Net Income Stabilization Account (NISA) program 
was established, under the Farm Income Protection Act, to help 
producers stabilize their income. The program enabled producers to 
accumulate funds in good years and to use those funds in bad years. As 
a complement to NISA, the Department introduced the Agriculture 
Income Disaster Assistance (AIDA) program in 1998 and replaced it 
with the Canadian Farm Income Program (CFIP) two years later. Both 
AIDA and CFIP were intended to help producers who experienced 
sudden and severe declines in farming income for reasons beyond their 
control. The CAIS program was introduced in December 2003. 


Problems in farm income support programs identified in our past audits 


4.3 
In 1996, we assessed the Department’s progress in implementing our 


We audited farm income support programs in 1991 and 1994. 


recommendations. Our audits questioned whether the Department 
had adequately measured and reported the effectiveness of the farm 
safety net programs and whether it had implemented proper financial 
management and controls. In particular, we noted that the 
Department 


* needed to strengthen program administration in areas such as 
payment accuracy verification, error correction, and on-farm 


audits; 


* needed to define program objectives and develop performance 
indicators; and 


* had not issued annual reports and financial statements in a timely 


manner. 
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The Canadian Agricultural Income Stabilization program 


44 The Canadian Agricultural Income Stabilization (CAIS) program 
aims to help producers protect their operations from small and large 
drops in income due to circumstances beyond their control. Producers 
can apply for CAIS support if they have farmed for more than six 
months and have reported farming income for income tax purposes. 


4.5 The Department delivers the CAIS program in all provinces and 
territories except Alberta, Ontario, Quebec, and Prince Edward Island, 
where the provincial governments deliver the program. In the 2005-06 
fiscal year, the Department spent about $1.1 billion on the CAIS 
program Canada-wide. The federal government is responsible for 

60 percent of the total amount spent on the CAIS program. The 
provincial and territorial governments are responsible for the 
remaining 40 percent. The federal government has signed bilateral 
implementation agreements with the provinces and territories that 
outline the cost-sharing and other terms and conditions for programs 
and activities, including the CAIS program. 


46 Producers receive a payment from the CAIS program when the 
farm income from their current year is less than their average farm 
income from previous years. 


47 The CAIS program is based on margins: 


¢ The production margin is allowable income minus allowable 
expenses for a given year. 


¢ The reference margin is the average production margin for three 
of the past five years; the years with the lowest and highest 
production margins are not included in the average. 


48 Producers receive payments when their current year’s production 
margin is less than their reference margin. The proportion of the 
government payment increases as a producer’s loss deepens—the 
greater the loss, the larger the government contribution. The payment 
amounts that producers receive vary widely. For example, in the 2004 
program year, where the Department delivered the CAIS program, it 
received 57,000 valid applications. Of the producers who submitted 
these applications 


¢ about 25,000 received no payment, 
¢ about 15,000 received $10,000 or less, 
* about 50 received payments of $500,000 or more, and 


¢ 12 received more than $1 million. 
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49 The Department estimates that accountants, and other third 
parties who charge a fee for the service, prepare 70 to 80 percent of the 
applications. The Department’s records indicate that it received about 
55,000 valid applications for the 2005 CAIS program year, compared 
to 57,000 and 55,000 applications for 2004 and 2003 program years, 
respectively. Producers can opt out of the CAIS program, but they are 
then ineligible to rejoin the program for two years. 


Focus of the audit 


4.10 In 2005, the House of Commons Standing Committee on 
Agriculture and Agri-Food asked the Auditor General to audit federal 
spending on the CAIS program. 


411 The purpose of the audit was to assess how well the Department 
* processes CAIS applications, 


* meets the monitoring requirements of the federal-provincial- 
territorial agreements, and 


* measures and reports its program performance to Parliament and 
other interested parties. 


4.12 More details about the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


4.13 The Financial Administration Act and the Treasury Board’s 
policies govern departments’ effective delivery and administration of 
transfer programs, such as the Canadian Agricultural Income 
Stabilization (CAIS) program. 


414 We expected the Department to have a system in place to ensure 
that it processes claims accurately and reasonably quickly. We also 
expected it to have adopted a reasonable approach to verify the 


completeness and accuracy of source data, an approach that 


¢ balances efficiency and effectiveness with the costs and risks to 


applicants and the federal government, and 


* is consistent with government requirements. 
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How the Department calculates benefits is not transparent to producers 


4.15 In their applications for income support, producers are required 
to submit income tax information. They are also required to submit 
complicated, detailed, and accurate information about farm expenses, 
revenues, and inventories each year. 


416 The method of calculating CAIS program benefits is 
complicated. The Department provides information to producers on 
its CAIS program website. It also provides group information sessions 
to producers as well as individual sessions, during which CAIS 
representatives advise producers on how to complete their 
applications. Despite these efforts, some producers have complained 
that 


° they do not understand how the Department calculates the 
benefits, 


¢ the benefits are not predictable, and 


¢ banks will not consider any potential payments from the CAIS 
program when deciding whether to lend them money. 


417 Stakeholders told us that the CAIS program payments are 
neither transparent nor predictable. The application forms do not give 
producers an idea of the amount they will receive. Producers provide 
the requested information and must wait until the Department has 
finished processing the application before they find out whether they 
will receive a payment. The Department provides an online CAIS 
benefit calculator on its website but does not guarantee the accuracy of 
the results. 


4.18 We expected the Department to provide producers with 
information that would help them to submit accurate information in 
their CAIS applications. To assess the Department's efforts to explain 
how staff members calculate payments, we reviewed the Department's 
external guidance materials, publications, notices, and presentations to 
applicants as well as the Calculation of Program Benefits statement. We 
also examined the Department's internal documents that are designed 
to guide staff members on how to communicate with producers. 


4.19 Some producers find it challenging to provide all of the complex 
information that the Department needs to process applications. 
Processors sometimes change the information in producers’ 
applications. For example, the manual requires processors to 
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* contact producers, for clarification, when producers use 
inaccurate codes to identify their commodities and/or inaccurately 
estimate the quantity of unsold inventory; 


¢ change commodity prices when producers submit prices that do 
not match the Department’s estimated market prices; and 


¢ change the information producers submit to reconcile the 
current year’s opening inventory with the previous year’s closing 
inventory and to make it match the data from Production 
Insurance (another federal-provincial-territorial agricultural 
program). 


4.20 We reviewed a sample of 2004 program-year applications to 
determine whether the Department informed producers of the changes 
that the processors had made to the applications. We also reviewed the 
Department’s guidance materials, which describe procedures for 
contacting producers. 


4.21 We found that, in some cases, the manual that is given to 
processors, for verifying the accuracy of the information producers 
submit, instructs staff to contact producers for additional information 
needed to process applications. However, we found that the 
Department’s guidance materials did not instruct staff on how to let 
producers know 


© how the revised information was used to calculate the benefits, 


what impact the changes would have on the benefits, 


what other changes they made to producers’ applications, or 


when these changes occurred. 


4.22 Changes not reflected in statements. The Calculation of 
Program Benefits statement lists, line-by-line, the figures that the 
Department used to calculate CAIS benefits, and is generally eight 
to ten pages long. However, it does not explain which figures the 
Department changed or why. 


4.23 By not explaining, in the Calculation of Program Benetits 
statement, the changes it makes to the figures in producers 
applications, the Department is missing an opportunity to build a 


well-informed and knowledgeable community of applicants. 


4.24 Onus on producers to question figures. Once they receive 
their Calculation of Program Benefits statement, producers have 
90 days to ask the Department to adjust the figures it used to process 


their applications or to request an appeal. Producers can also call the 
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Department’s CAIS toll-free line to discuss the figures the Department 
changed and the reasons for the changes. 


4.25 In our opinion, because the statement does not indicate which 
figures were changed, producers may not notice that changes have 
been made. The onus is on producers to compare the numerous figures 
in the final eight- to ten-page statement with the figures they 
submitted in their original applications. 


4.26 If the Department clearly indicated the changes, explained the 
reasons for those changes, and explained how those changes affected 
the benefit amounts, the process would be more clear, transparent, and 
fair for the producers. It would also improve the producers’ ability 


* to correct figures that the Department had changed incorrectly, 


¢ decide whether to ask the Department to adjust the figures used 
to calculate the benefit, 


* appeal the Department's decision, and 


* provide more accurate information on future CAIS applications. 


4.21 Transparency needs to improve. During our review of a sample 
of the 2004 program-year applications, we found one payment made to 
a producer that increased by more than $90,000 due to a change in 


processing procedures. This change occurred after almost 20 percent of 


the applications for the 2004 program year had already been processed. 
The Department told us that it did not specifically inform producers, 
whose applications had already been processed of this change, nor did 
it inform them of the effect it may have had on their program benefits. 
However, the Department did tell us that it would investigate this 
situation to determine if proper processes were followed and that it 
would address any issues it identifies. Nevertheless, we are concerned 
that these producers may not have known of the changed procedures 
and, therefore, would not have had appropriate information to decide 
whether to request an adjustment or launch an appeal. 


4.28 We also noted that in its guidance documents for producers the 
Department does not mention that it will accept re-assessments by the 
Canada Revenue Agency. If these re-assessments affect a producer’s 
CAIS reference margin, the Department will accept the new 
information for up to three years after the end of the program year 
affected by the adjustment. The Department said it would mention 
this practice in the next update of the CAIS handbook for producers. 


4.29 The Department plans to improve the CAIS application forms 
and tools. By the time this report is tabled, it intends to simplify the 
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application form for the 2006 CAIS program year, by reducing the 
amount of information required, and to provide a web-based electronic 
CAIS application tool and online access to CAIS account information 
for individual producers. 


4.30 As we have already noted, the CAIS program is complicated. 
The producers are responsible for providing accurate information 

in their CAIS applications; the onus is also on the producers to 
question figures and identify incorrect payments. Expanding the 
information in the final Calculation of Program Benefits statements 
would improve producers’ ability to detect incorrect figures and decide 
whether to ask the Department to adjust the calculations or to appeal 
the Department’s decision. Expanding the information would also be 
more consistent with the Department's objective of client service 
excellence and with the primary objective of the CAIS program, as set 
out in the Farm Income Protection Act, which is to protect producers’ 
incomes. 


4.31 Recommendation. Agriculture and Agri-Food Canada should 
improve the transparency of the final statement that it sends to 
producers about their farm income support program benefits. The 
Department should clearly and fully explain how it calculates benefits, 
including what information it changes and why it makes the changes. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation and has already taken steps to 
improve the transparency of the information it provides to producers. 
Mote specifically, it intends to add explanations of benefit calculations 
and adjustments and what to do if producers have questions or 
disagree with the Department’s decisions to the Calculation of 
Program Benefits statements for the 2007 year. Implementing this 
recommendation will also add to improvements that the Department 
has already made to its communications with producers as well as to 
changes to make the program simpler and more accessible, such as 
making program parameter changes, harmonizing its forms with the 
Canada Revenue Agency, reducing its supplementary form, and 
offering on-line My Account and calculator features. 


432 Controls to ensure accurate payments. We expected the 
Department to have implemented controls to ensure the accuracy 


of CAIS payments, and we reviewed the controls. 


433 The Farm Income Programs Directorate is responsible for 
processing CAIS applications. It is also initially responsible for 


approving payments. The Financial Operations Division 
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Absolute errors—the total dollar value of 
overpayment errors added to the total dollar 
value of underpayment errors. 


independently reviews the Directorate’s recommended payments. If 
the payments are accurate, the Division approves them. This review is 
the last step taken to verify that payments are accurate, before the 
government sends cheques and the Calculation of Program Benefits 
statements to producers. 


4.34 The Department uses a risk-based verification strategy to 
process CAIS applications. Based on the risk assessment results, the 
Farm Income Programs Directorate processes about two thirds of the 
applications manually. It processes the rest electronically, with no 
human intervention after the initial data assessment stage. After 
payments have been made, the Financial Operations Division also 
reviews samples of payments to estimate absolute errors. 


4.35 Payment errors declining. The Department’s goal is to ensure 
that the total estimated absolute errors to producers amounts to 
three percent. For the 2003 and 2004 program years, this would have 
represented about $20 million per year. 


4.36 However, it has so far been unable to meet this target: 


¢ For the 2003 program year, the Financial Operations Division 
estimated that absolute errors in payments to producers totalled 
$70 million, or 10.2 percent. 


¢ For the 2004 program year, the Division estimated that absolute 
errors in payments to producers totalled $43 million, 
or 6.5 percent. 


4.37 When we completed our audit, the Division’s preliminary 
calculations for the 2005 program year indicated that estimated 
absolute errors were lower than in previous years and were beginning 
to approach the internal target of three percent. 


4.38 Pre-payment reviews. The Financial Operations Division 
reviews all applications for payments of more than a specified amount 
before approving them. It then returns applications with payment 
errors to the Directorate, so the errors can be corrected. 


4.39 All payments that exceed the specified amount must be included 
in the Financial Operations Division’s pre-payment reviews. The 
number of pre-payment reviews that the Division does depends on 
how much confidence it has in the accuracy of the Farm Income 
Programs Directorate’s application processing. By the time we 
completed our audit, the Division’s records showed that it had 
conducted pre-payment reviews of about 12,000, or more than 

50 percent, of the 2005 program-year applications that resulted in 
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payments to producers. It returned just over 4,000, or about 

30 percent, of the applications it reviewed to the Directorate for 
correction. These applications represented a pre-payment error 
of 7.2 percent, or pre-payment errors totalling about $33 million. 


4.40 Post-payment reviews. To assess the Department’s overall 
processes and estimate errors in payments, the Financial Operations 
Division also reviews samples of payments after the cheques and 
Calculation of Program Benefits statements have been sent to 
producers. From these reviews, the Division estimates that 65 percent 
of the dollar value of errors in payments made to producers is due to 
overpayments, and 35 percent is due to underpayments. For the 2004 
program year, this would have resulted in estimated overpayments of 
about $28 million and estimated underpayments of about $15 million. 


441 The Financial Operations Division also uses information from 
the post-payment reviews to determine the specified amount for the 
pre-payment reviews. To reduce the errors in payments, the Division 
lowered the specified amount for its reviews each CAIS program year. 
Because more payments are now subject to review, the Division has 


had to increase its staff—from 36 in 2005 to 42 in 2006. 


4.42 The Financial Operations Division’s role is to assess whether the 
Farm Income Programs Directorate has adequate procedures and 
whether those procedures are working well. The Division’s workload is 
directly dependent on how accurately the Directorate calculates 
program payments. The Division is detecting a large number of errors, 
representing a high dollar value, after applications have been processed 
by the Directorate. In our opinion, it would be more efficient for the 
Directorate to improve its procedures to prevent errors in processing, 
rather than relying on the Division to detect them during their reviews. 
at the end of the process. 


4.43 As mentioned earlier, the Financial Operations Division uses 
information it gathers from the post-payment reviews to estimate the 
absolute errors in payments to producers. We found that the Division 
uses a reasonable methodology for estimating errors. However, it does 
not have the statistical support to determine the accuracy of the 


estimate. 


4.44 Recommendation. Agriculture and Agri-Food Canada should 
use a statistically valid methodology to estimate errors in farm income 


support program payments. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation and is investigating the cost and 
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feasibility of implementing additional accuracy determination 
methodologies to supplement approaches, which it will implement if 
they are found to be cost-effective and feasible. Added to the 
significant progress the Department has already made to reduce errors, 
implementing this recommendation could further enhance its current 
approach to internal reviews and verification. 


4.45 Dollar amounts of errors not publicly reported. The 
Department’s target for errors in payments to producers is 

three percent. The Department does not report this target, or how 
well it meets it, to Parliament, stakeholders, or other interested parties, 
even though this information would be helpful in assessing the 
Department’s performance. 


4.46 Recommendation. Agriculture and Agri-Food Canada should 
establish and publicize a target for the accuracy of payments for its 
farm income support programs, and should report to Parliament, and 
other interested parties, how well it meets this target. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with establishing a target for the accuracy of payments and, as 
noted, has in fact established an internal target. The Department will 
also take into consideration the recommendation to report to 
Parliament, and other interested parties, on its achievements in 
meeting payment accuracy targets. 


The Department focuses on minimizing overpayments 


447 The primary objective of the CAIS program is to help producers 
protect their farm income from risks, such as adverse weather, diseases, 
and low market prices. It is a complicated program that requires 
producers to provide accurate and detailed information in their 
applications about expenses, revenues, and inventories. For example, 
when completing their applications, producers have to select the 
correct codes from a list of more than 1,500 different codes. 


4.48 Because this complexity can lead to errors in the information, 
we expected the Department to have processes to equitably manage - 
the risk of underpayments and overpayments to producers. The 
Department’s approach for two previous farm income support 
programs, the Agriculture Income Disaster Assistance (AIDA) 
program and the Canadian Farm Income Program (CFIP), focused on 
both overpayments and underpayments, with full verification and 
review of all payments. Therefore, we assessed how the Department 
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processes applications and ensures that applicants receive the benefits 
they are eligible for. 


4.49 CATS payment errors can also affect payments from other 
programs, because the Department sometimes uses producers’ CAIS 
data to calculate their benefits for other programs. 


4.50 In March 2006, officials told the Department’s senior 
management that one of the important elements of the Department’s 
approach to the CAIS program is the “focus on overpayment risk.” The 
Department told us that it detects and corrects a large number of errors 
before payments are sent to producers and that its approach reflects 
producers’ interests and the overall integrity of the CAIS program. 


4.51 The Department also told us that it takes a balanced approach to 
overpayments and underpayments. Nevertheless, we found a number 
of instances where the Department’s processes and controls place more 
focus on identifying and correcting overpayments compared to 
underpayments. 


4.52 We found that one of the principal ways the Department focuses 
on overpayments is through the risk assessment tests. The Farm 
Income Programs Directorate reviews applications in detail when the 
major risk assessment tests indicate a higher than expected payment 
and, therefore, a risk of overpayment to the producer. However, the 
risk of an underpayment to the producer does not trigger a similarly 
detailed review. The Directorate then sends the payment files to the 
Financial Operations Division for independent review before the 
cheques are sent to producers. 


4.53 The Department does not dispute the fact that the Farm Income 
Programs Directorate’s major risk assessment tests focus on the 
overpayment risk. Nor does it dispute the fact that the Financial 
Operations Division’s reviews represent the last step in the process to 


verify that payments are accurate. 


4.54 However, according to its pre-payment review criteria, the 

Financial Operations Division does not review applications that have 
payments that are below the specified amount. It also does not review 
“zero payments,” which represent about 45 percent of all applications. 
Consequently, about 70 percent of the applications filed each program 
year are not reviewed by the Division before the Department sends the 
Calculation of Program Benefits statements to producers. If a payment 
is an underpayment and is below the specified amount, the application 
would not be reviewed by the Division before the producer receives the 


Calculation of Program Benefits statement. In our opinion, the 
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payment of any amount is important to producers. In addition to 
paying a fee for participating in the CAIS program, 70 to 80 percent of 
producers also pay fees to have their applications prepared. 


4.55 Weare concerned that the Department's pre-payment processes 
do not control the risk to producers of underpayments for files below 
the specified amount that are not reviewed by the Financial 
Operations Division. The Department puts the onus on producers to 
question the figures and identify incorrect payments. However, the 
Department does not believe that what we consider to be a focus on 
overpayments has a significant impact on producers. 


456 The Department estimates the absolute errors in payments to 
producers, which include both underpayments and overpayments. 
However, we believe two additional steps need to be taken to confirm 
this estimate. As we recommend earlier, the Department should use a 
statistically valid methodology to estimate errors in farm income 
support payments. 


4.57 The Department also needs to include zero payment applications 
in its samples for post-payment reviews. By their nature, zero payment 
applications only have a risk of underpayment, not overpayment. 
About 45 percent of all applications result in zero payments. If these 
zero payments are excluded from the post-payment samples to detect 
errors, there is a risk that the Department’s estimate that only 

35 percent of the errors are underpayments is inaccurate. 


458 Recommendation. When estimating errors in payments from 
farm income support programs, Agriculture and Agri-Food Canada 
should include zero payment applications in its post-payment samples. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation and is investigating the cost and 
feasibility of implementing additional accuracy determination 
methodologies, including sampling zero payment files, which it will 
implement if they are found to be feasible and cost-effective. 


459 Another example of the Department’s focus on overpayments is 
the processing instructions for cases where producers provide a 
plausible reason for the Department to accept new information after 
the 90-day deadline for requesting changes or launching appeals. 
These instructions direct processors to “determine the correct values, 
update the . . . information, and reprocess . . . only where the payment 
will decrease.” There is no equivalent instruction for underpayments. 
However, the Department informs us that the federal-provincial- 
territorial agreements limit adjustments that affect payments to 90 
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days from the date of the Calculation of Program Benefits statement. 
The federal-provincial-territorial agreements also require the 
Department to recover overpayments whenever they are discovered. 


4.60 Recommendation. When processing applications for benefits, 
under farm income support programs, the emphasis that Agriculture 
and Agri-Food Canada places on efforts to prevent, detect, and 
correct both under-payments and over-payments to producers should 
be proportional to the relative risks of underpayments compared with 
overpayments—as determined through the Department’s revised 
post-payment sampling results—as recommended elsewhere in 

this chapter. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation. It will undertake, as indicated in 

its responses to other recommendations elsewhere in the chapter, to 
adjust its review and verification processes; to ensure a balance 
between overpayments and underpayments; and to ensure that, by 
the time payments reach producers, both have been identified and 
corrected to the same degree. Efforts in this regard will be proportional 
to the relative risks of underpayments compared with overpayments 
and will reflect both producers’ interests and the overall integrity of 
the program. 


The Department needs to strengthen its focus on continuous improvement 


4.61 We expected the Department to have developed and 
implemented procedures to monitor and improve the delivery of the 
CAIS program. To determine whether it maximizes opportunities to 
improve processing accuracy, we reviewed the way the Department 
analyzes and uses data. 


4.62 Data not used well. We found that a number of groups in the 
Department collect a lot of useful information, but the Farm Income 
Programs Directorate is not systematically using that information to 


improve processing accuracy. 


463 The Directorate’s system for processing applications includes 
three quality assurance reviews, at different stages, for which reports 
are sent to the Directorate’s processing managers. These reports focus 
on the number of applications containing errors that were returned for 
processors to correct and the number of applications that are still 
outstanding. However, we found that the Directorate did not analyze 
the data to identify trends or target specific areas for monitoring. 


Chapter 4 15 


CANADIAN AGRICULTURAL INCOME STABILIZATION—AGRICULTURE AND AGRI-FOOD CANADA 


4.64 As we indicated earlier, by the time we completed our audit, the 
Financial Operations Division had reviewed more than 50 percent of 
the 2005 payment applications and had returned about 30 percent to 
the Farm Income Programs Directorate for correction. Through these 
reviews, the Division gathers important data about processing errors 
as well as information about the errors producers make in their 
applications. However, the Division and the Directorate sometimes 
disagree about what constitutes an error and which errors require 
correction. We also found that the Directorate does not systematically 
use or analyze the Division’s data to improve the accuracy of its 
application processing. 


4.65 Furthermore, the Farm Income Programs Directorate’s audit 
services unit conducts on-farm audits of producers’ records to verify 
the accuracy of payments made to producers. At the time of our audit, 
the unit was completing the on-farm audits for the 2003 program year 
and was beginning the audits for the 2004 and 2005 program years. 


4.66 We found some inaccuracies in the database the Department 
uses to record the results of the on-farm audits. Nevertheless, the 
Department's analyses of the information in the database indicate that 


* about 90 percent of the audits of 2003 program-year payments 
resulted in changes to the original payment amount, and 


* about 30 percent of the audits changed the payments by $5,000 or 
more. 


4.67 We also found that the auditors do not analyze the causes of 
the errors—whether the Department made errors processing the 
application or whether producers submitted inaccurate information. 
In our opinion, these audits could provide valuable information about 
the accuracy of the data submitted by producers and about the 
accuracy and reliability of the Department’s processing procedures. 


4.68 Despite the numerous sources of relevant information, the 
Directorate does not systematically gather or analyze the data. It is 
therefore missing an opportunity to continuously improve the way 
it processes CAIS applications. 


Some improvements are needed in monitoring and training processing staff 


4.69 Although it has automated some steps, the Farm Income 
Programs Directorate still relies heavily on its staff to manually process 
the applications that fail the major risk assessment tests. Therefore, the 
employees that process applications need appropriate guidance and 
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training to do their work accurately. We reviewed the Department’s 
training materials and the way it monitors its processors’ performance. 


410 We found that the Department appoints senior processors as 
resource people to offer guidance to other processors. However, we also 
found that processors do not have to have successfully completed 
training before they begin processing applications, and the Department 
has not established a grading system for the training quizzes that are 
supposed to test how well employees understand the material. Some 
employees do not take the quizzes, some do poorly on the quizzes, and 
some are absent for significant portions of the courses. 


4.71 Processing managers use quality assurance reports to monitor 
processors’ performance and to determine whether they follow 
procedures. The processing managers ask the quality assurance team to 
increase the monitoring if processors make too many procedural errors. 


472 The quality assurance reviews could also be used to provide 
more prompt and useful feedback on individual processors’ errors, so 
managers can better decide what additional training employees need. 


473 In addition, while we found that processors’ individual 
performance objectives include targets for production, timeliness, and 
adherence to procedures, there are no targets for payment accuracy. 
By the time we completed our audit, from its pre-payment reviews, the 
Financial Operations Division had returned about 30 percent of the 
applications to the Farm Income Programs Directorate for correction. 
These applications represented a pre-payment error of 7.2 percent, or 
pre-payment errors totalling about $33 million. 


4.74 Recommendation. Agriculture and Agri-Food Canada should 
strengthen its focus on continuous improvement in the administration 
of its farm income support programs by systematically 


* collecting relevant information about the nature, extent, and 


timing of errors; 
* analyzing the information; and 


* continually applying the results of the analyses to improve all 


phases of application processing. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation to strengthen its focus on continuous 
improvement. Specifically, the Department agrees that using all the 
information it has available more systematically will ensure that it 
builds on past improvements, while fostering a stronger continuous 


improvement culture. 
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Service time standards are not being met 


4.75 The Department aims to deliver efficient, effective, client- 
focused programs. However, we found that one of the common 
complaints from producers is that CAIS payments are not timely. 


4.16 We reviewed a sample of the 2004 program applications and 
calculated how long it took the Department to process them—from 
the date the Department received an eligible application to the date 
it sent the Calculation of Program Benefits statement to producers. 
We found that, on average, the Department took 120 days to process 
the applications. 


4.11 During the first two years of the CAIS program, the Department 
measured its service time standard from the date the application was 
entered in the database to the date the application for payment was 
approved. The Department aimed to process applications received 
within 60 days during non-peak times and within 75 days during peak 
periods. The Department's records indicate that it met its service 
standard 65 percent of the time in 2003 but only 26 percent of the 
time in 2004. 


4.18 For the 2005 program year, the Department began to deduct the 
time that processors spend waiting for producers to provide additional 
data from the time taken to process an application. As a result, the 
Department expects its performance against its service standard to 
improve. During our review of a sample of the 2004 program 
applications, we found that, most of the time, processors requested 
additional information by telephone and usually received it within 


24 hours. 


4.19 The Department also increased its service time standard to 

90 days during peak periods. In addition, it now counts the days from 
the date it receives an eligible application to the date it approves a 
claim for payment (and considers processing complete). At the end of 
our audit, the Department’s records indicated that it met its service 
time standard for the 2005 program-year applications, about 

38 percent of the time. 


4.80 We agree that using the date the Department receives the 
application from a producer instead of the date the information is 
loaded into its database is an appropriate way to measure how long it 
takes to process applications. However, a more client-focused approach 
is to use the date the Department sends Calculation of Program 
Benefit statements to producers—when it informs them of the 
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outcome of their application—instead of the date an application is 
approved for payment. 


481 The Department instructs processors to keep diligent records 
of all contact with producers in the electronic database, including the 
dates they requested and received additional information. We agree 
that accurately measuring how long it takes to process applications 
depends on processors recording accurate and complete information. 
We also agree with the Department’s opinion that there may be a 
trade-off between accuracy and timeliness. While taking more time 
to review applications can reduce errors, it can also increase the time 
producers have to wait for their benefits. 


4.82 Recommendation. Agriculture and Agri-Food Canada should 
set realistic service time standards for processing farm income support 
program applications and should accurately measure its performance 
against those standards. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the recommendation and will continue working with 
provincial partners to establish national service standards for the 
CAIS program to improve the predictability of CAIS processing, 


The Department is not fully meeting the audit and reporting requirements of 
the agreements 


483 The federal-provincial-territorial agreements include important 
monitoring provisions that enable each of the parties to confirm that 
the Canadian Agricultural Income Stabilization (CAIS) program is 
being administered according to the agreements’ terms and conditions. 
Alberta, Ontario, Quebec, and Prince Edward Island deliver the CAIS 
program to their own provinces; the federal government delivers it to 
the rest of Canada. 


4.84 Given that the federal government pays 60 percent of the CAIS 
program-delivery costs for the provinces and territories and that the 
provinces and territories pay the other 40 percent, both levels of 
government need to know whether funds are being properly managed 
and whether the program is being delivered according to the 
federal-provincial-territorial agreements. 


485 We expected Agriculture and Agri-Food Canada to have 
mechanisms in place to ensure that all parties are meeting the 
monitoring provisions—the auditing and reporting requirements—of 
the agreements. We assessed how well the Department ensures 


compliance with the audit and reporting provisions in the CAIS 
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agreements. We did not audit the provinces’ delivery of the CAIS 
program. 


4.86 The agreements that govern the CAIS program include various 
audit provisions. One audit requirement in the agreements addresses 
financial reconciliations and states that all parties must 


* reconcile their claims for program payments and administrative 
expenses with the amounts reported in their audited financial 
statements, 


¢ conduct audits of those reconciliations, and 


° provide the audits to those who made the payments within nine 
months of the end of the fiscal year. 


4.87 We found that the Department had not prepared its 
reconciliation audits by the deadlines, but had completed them by the 
time we completed our audit. We also found that the Department did 
not obtain any of the audited reconciliations of the provincial agencies’ 
claims within the deadlines. However, by the time we completed our 
audit, all but two of the four audits were complete. The Department 
took appropriate action to obtain the required information from the 
other two provincial agencies, eventually withholding federal funding 
from one of them. 


4.88 Another audit provision in the agreements requires the 
Department to periodically conduct compliance audits, at its 
discretion, to determine whether the federal and provincial agencies 
administer the CAIS program according to the agreements’ terms and 
conditions. Compliance audits also enable the Department to assess 
national consistency in program delivery. We found that the 
Department had not completed any compliance audits, although it 
was planning one for the 2004 program year that would include all 
provincial agencies as well as the Department. 


4.89 The agreements also require each administration to prepare an 
annual report on their CAIS operations, revenues, and expenditures 
as soon as possible following the end of each fiscal year. We found that 
neither the Department nor the provincial administrations had 
completed any of the required reports. 


490 Recommendation. Agriculture and Agri-Food Canada should 
ensure that it meets the audit and reporting requirements of the 
federal-provincial-territorial agreements. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with the need to meet the audit and reporting requirements 
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of the federal-provincial-territorial agreements. In this regard, it has 
already been working with provinces on finalizing audit parameters 
and carrying out audits and will accelerate this work to ensure that 
audits are conducted in a more timely manner. 


Performance monitoring and reporting needs to be improved 


4.91 According to the Treasury Board’s policies and guidelines (the 
Policy on Transfer Payments and its guide, Preparing and Using Results- 
based Management and Accountability Frameworks), departments need 
to establish mechanisms that enable managers to measure and report 
program results and to assess program effectiveness. We expected 
Agriculture and Agri-Food Canada to have developed a framework for 
measuring the effectiveness of the CAIS program and for reporting to 
Parliament and other interested parties. 


4.92 We found that the Department has developed a results-based 
management and accountability framework that contains 
performance indicators for all its Business Risk Management 
programs—including the CAIS program—and established high-level 
expected results for those programs. However, for the CAIS program, it 
has not established specific and measurable targets to assess program 
performance or set deadlines to develop these targets. In our view, the 
lack of specific and measurable targets reduces management's ability to 
assess whether the CAIS program is achieving its objectives and to 
report to Parliament on the performance of the CAIS program. 


4.93 Furthermore, while it has established performance indicators in 
the Results-Based Management and Accountability Framework, the 
Department is not preparing timely, comprehensive reports on all of 
them. For example, the data the Department routinely gathers focuses 
on operational information, such as the number of CAIS applications 
it received and processed and the amounts it paid producers. When we 
completed our audit, the Department had only prepared a 
comprehensive report for the 2003 CAIS program year. 


4.94 While it continually updates this report as new information 
becomes available, the Department told us that it needs to wait until 
it has processed all 2004 CAIS applications, before it can prepare a 
similar report and measure its performance against the CAIS program’s 
performance indicators for that year. 

4.95 By the end of October 2006, the Department was still processing 
830, or 1.5 percent, of the 57,000 applications it had received for 


the 2004 program year. In our view, monitoring program performance 
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Conflict of interest 


against the key indicators need not wait until the Department has 
processed all the applications. Even if all of the necessary information 
is not yet available, the Department can prepare comprehensive 
reports for each CAIS program year, which would be useful for 
monitoring and improving the delivery of the CAIS program and for 
reporting CAIS program results to Parliament. 


4.96 Recommendation. Agriculture and Agri-Food Canada should 


* establish specific and measurable targets for all of the performance 
indicators in its Results-Based Management and Accountability 
Framework; and 


* produce timely reports to Parliament on the performance of farm 
income support programs, measured against those targets. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees with this recommendation. It will, in conjunction with its 
provincial counterparts, develop targets for each of the measures in 
the Results-Based Management and Accountability Framework and 
report progress and achievements, on these targets, in the annual 
departmental performance report to Parliament. This task will be 
completed by December 2007 and will include, for example, targets 
related to producer participation in CAIS and to timeliness of 
payments. 


The Department needs to continue its efforts to avoid conflicts of interest 


491 We reviewed potential conflict of interest cases, in which 
employees who process applications for farm income support payments 
help producers prepare their applications. In our opinion, processors 
who also work as private consultants to help prepare Canadian 
Agricultural Income Stabilization (CAIS) applications contravene 
conflict of interest provisions in the Values and Ethics Code for the 
Public Service. These employees also risk contravening provisions 

that deal with providing preferential treatment and benefiting from 
information that is not available to the public. When we brought 
these cases to Agriculture and Agri-Food Canada’s attention, it began 
investigating them. 


4.98 The Department assesses CAIS applications against risk 
assessment tests. Applications that pass the tests are deemed low risk 
and are less likely to be reviewed by the Farm Income Programs 
Directorate. Individuals with detailed knowledge of these tests could 
use that knowledge to manipulate the data in producers’ CAIS 
applications and to trigger payments. 
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4.99 We are not aware of any cases where such manipulation actually 
occurred. However, it may occur if CAIS employees help to prepare 
CAIS applications (as private consultants). 


4.100 The Department told us that only employees who develop and 
implement these tests have full access to the calculations and details 
on how the tests work. The Department also told us that the tests, and 
the way processors use them to assess applications, are so complex that 
they would not be useful to employees. 


4.101 We found that the processors’ manuals contain specific 
explanations for why data from producers fail the tests. The 
Department requires processors to understand how the test result was 
calculated and to explain any possible anomalies. If they conclude that 
the level of risk associated with the application is acceptable, they can 
send the application “straight to payment.” 


4.102 Department officials also told us that some CAIS processors 
prepare CAIS application forms for their own farms. Processors must 
indicate that they are Department employees to ensure that their 
applications, and those of their immediate colleagues, are processed 
by other teams. 


4.103 The Department sends annual notices to employees reminding 
them of their responsibilities under the Values and Ethics Code for 

the Public Service. After we notified the Department of the potential 
conflict of interest cases, it sent an additional reminder to 18 of 

the almost 400 employees who process applications for farm income 
programs. 


4.104. We reviewed the confidential reports that several employees filed 
as a result of this reminder as well as their past reports and reports 
previously filed by other employees. Some CAIS employees said 

they were providing private consulting services on preparing CAIS 


applications, and some said they were charging fees for their services. 


4.105 Some of the employees had disclosed such activities to 
management before we raised the issues with the Department— 

of those, some had done so for a number of years. However, the 
Department told us that it did not have records of follow-up action 
for all of the confidential reports that had been filed. After we raised 
our concerns about potential conflicts of interest, the Department 
conducted investigations and told five employees to stop preparing 


CAIS applications for their clients. 
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4.106 We also found that the Department had not always properly 
protected documents containing information about the risk assessment 
tests, as required by the Government Security Policy. However, as 

the Department does currently identify recent versions of these 
documents, if any of the almost 400 employees of the farm income 
programs processing unit were to use this information for private 


purposes, they would contravene the Policy and the Values and Ethics 
Code for the Public Service. 


4.107 Finally, we found that although applicants may voluntarily 
disclose the name of the individual or company that prepared the 
application, they are not required to do so. 


4.108 The Department informed us that it was strengthening its values 
and ethics regime, before we notified them of potential conflicts of 
interest. Furthermore, in December 2006, it reminded all employees of 
their obligations, under the Values and Ethics Code for the Public Service. 
It specifically told employees to file a report if they 


* owned a farm and were applying for income assistance from the 
Department, or 


* were considering helping producers apply to the Department’s 
grant and contribution or farm income support programs. 


4.109 The Department also told us that it is implementing a process to 
address risks associated with employees preparing applications for their 
own farms, which would include on-farm audits if necessary. 


4.110 Recommendation. Agriculture and Agri-Food Canada should 
strengthen measures to reduce the risks of potential conflicts of 
interest. At a minimum, the Department should strengthen its values 
and ethics program—making it clear to employees that they cannot 
prepare applications for others for a fee—and establish processes to 


¢ identify processors who also prepare applications for farm income 
support programs; 


¢ address risks associated with employees preparing applications for 
their own farms or their families’ farms, including performing on- 
farm audits if necessary; and 


* ensure that all applications prepared by farm income support 
program employees are approved manually rather than 
electronically. 


Agriculture and Agri-Food Canada’s response. The Department 
agrees. It has been strengthening, and will continue to strengthen, its 
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values and ethics program, by continuing to advise all new staff of their 
obligations, under the Values and Ethics Code for Public Servants, and 
reminding them of their obligations on an annual basis. Recently, the 
Department strengthened its conflict of interest processes to 
proactively involve management in the resolution of potential or 
actual conflicts of interest. The Department has also recently 
instituted a formal training program and will be delivering this program 
across the country. Additionally, it has instituted a 1-800 hotline 
number where employees and managers can get immediate assistance. 


CAIS processors are required to self identify if they are completing a 
CAIS application for themselves, for a family member, or for friends. 
They are also asked to identify that application, and these applications 
are then processed by someone other than themselves in a separate 
team designated for that function, effectively distancing the employee 
from that application. Finally, the Department has informed employees 
through a Deputy Minister letter that they cannot prepare CAIS 
applications for a fee. This will be repeated on a yearly basis. 


Conclusion 


4.111 Where Agriculture and Agri-Food Canada delivers the 
Canadian Agricultural Income Stabilization (CAIS) program, the 
Department does not have a system in place to assure itself that 
applications are processed accurately. The Department focuses on 
detecting and correcting overpayment errors; its method of estimating 
errors in payments lacks statistical support, and it omits the sampling 
of zero payment applications. Consequently, there is a risk that the 
impact of the focus on overpayment is greater than that estimated by 
the Department. However, the Department does not believe that there 
is a significant impact on producers in what we consider to be a focus 
on overpayments. The Department has improved its accuracy of 
processing 2005 applications that result in a payment to producer; 
however, while close to reaching its target for payment accuracy, the 
Department has not yet met it. 


4.112 Agriculture and Agri-Food Canada needs to improve the 
efficiency of processing applications and the effectiveness of managing 
the risks to applicants and to the federal government. Specifically, we 


found the following: 


¢ The way the Department calculates benefits is complex and not 
transparent. The Department does not indicate or explain to 
producers how and why it changes the figures they originally 
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submitted or how it uses the altered figures to calculate the 
producers’ final CAIS benefits. 


¢ The Department needs to improve its systematic use of available 
information to strengthen its continuous improvement efforts. 


¢ The Department focuses on detecting and correcting payment 
errors at the end of the process instead of focusing on preventing 
them in the first place. 


¢ There are long delays before producers are told whether they will 
receive a benefit. On average, the Department took 120 days to 
process 2004 program-year applications. 


4.113 Given the amount of complex information producers are 
required to provide in their CAIS applications and the volume of 
applications it receives, the Department has implemented a reasonable 
approach to verify the completeness and accuracy of source data. 


4.114 The Department is not meeting its service time standards for 
processing applications and, therefore, is not processing applications 
within a reasonable period of time. 


4.115 The Department has not established an external service 
standard for payment accuracy, nor has it established specific and 
measurable targets for all of the performance indicators in its 
Results-Based Management and Accountability Framework. 
Establishing such targets would improve the Department’s ability to 
assess the CAIS program, to determine whether it is achieving its 
intended objectives, and to report on its performance to Parliament. 
Furthermore, ongoing monitoring of the CAIS program’s performance 
against the key indicators would be useful for improving the delivery of 
the program and reporting its results to Parliament. 


4.116 With the exception of the reporting requirements, the 
Department substantively ensured that the monitoring requirements 
of the federal-provincial-territorial agreements were met. We did not 
audit the delivery of the CAIS program by the provincial 
administrations. 
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About the Audit 


Objectives 


Our first objective was to determine whether Agriculture and Agri-Food Canada has reasonable 
approaches to verify the completeness and accuracy of the source data for Canadian Agricultural Income 
Stabilization (CAIS) applications and to ensure that claims are processed accurately and reasonably 
quickly. 


Our second objective was to determine whether the Department has developed a performance 
measurement framework and is using it to 


* assess how well the CAIS program meets its objectives; 
* monitor and improve CAIS program delivery; and 


* report to Parliament and other interested parties. 


Our third objective was to determine whether the Department ensures that parties to the federal- 
provincial-territorial agreements that govern the delivery of CAIS are meeting monitoring requirements. 


Scope and approach 


The CAIS program provides income support to producers. It is administered by the Department for all 
provinces and territories except Alberta, Ontario, Quebec, and Prince Edward Island. Our audit focused 
on the processes and procedures that the Department uses to manage and deliver the CAIS program. 


Specifically, we examined how the Department 
* processes applications for income support, 
* measures and reports on its performance to Parliament, and 


* meets the monitoring requirements of the federal-provincial-territorial agreements. 
We did not audit the provinces’ delivery of the CAIS program. 


The stakeholders that we interviewed included officials from 
* several non-governmental organizations that represent more than 200,000 producers, 
* the major banks, and 


* provincial governments where the CAIS program is and is not delivered by the Department. 


We interviewed officials and reviewed documents in Winnipeg (where the CAIS program is administered) 
and at the Department’s national headquarters in Ottawa. We also reviewed files from a representative 
sample of 137 CAIS applications from the 2004 program year. The eligible 56,515 applications processed 
on or before 12 April 2006 were divided into low, medium, and high dollar amounts and zero-payment 
applications. Sample sizes from each of the four groups were selected, so confidence intervals for deviation 
rates would be less than 10 percent at a 90-percent confidence level. All estimates reported in the chapter 


have been weighted to accurately reflect the population. 
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Criteria 
We expected the Department to have 


- established policies and procedures to ensure efficient and effective delivery and administration of the 
CAIS program, 


- developed a results-based management and accountability framework that provides for appropriate 
measurement and reporting of results to the Department’s managers as well as to Parliament and other 
interested parties, 


- developed appropriate evaluation criteria to assess the effectiveness of the CAIS program, 
- developed and implemented processes to monitor and improve CAIS program delivery, and 


» ensured that all parties to the federal-provincial-territorial agreements had fulfilled their respective 
audit and reporting responsibilities regarding the CAIS program. 
Audit work completed 


Audit work for this chapter was substantially completed on 31 October 2006. 


Audit team 


Assistant Auditor General: Andrew Lennox 
Principal: Frances Taylor 
Director: Raymond Kunze 


Ian Campbell 
Alina Dan 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 4. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Canadian Agricultural Income Stabilization program applications 


4.31 Agriculture and Agri-Food 
Canada should improve the 
transparency of the final statement that 
it sends to producers about their farm 
income support program benefits. The 
Department should clearly and fully 
explain how it calculates benefits, 
including what information it changes 
and why it makes the changes. 
(4.15-4.30) 


444 Agriculture and Agri-Food 
Canada should use a statistically valid 
methodology to estimate errors in farm 
income support program payments. 


(4.32-4.43) 


4.46 Agriculture and Agri-Food 
Canada should establish and publicize a 
target for the accuracy of payments for 
its farm income support programs, and 
should report to Parliament, and other 
interested parties, how well it meets this 


target. (4.32-4.45) 
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The Department agrees with the recommendation and has 
already taken steps to improve the transparency of the 
information it provides to producers. More specifically, it intends 
to add explanations of benefit calculations and adjustments and 
what to do if producers have questions or disagree with the 
Department's decisions to the Calculation of Program Benefits 
statements for the 2007 year. Implementing this 
recommendation will also add to improvements that the 
Department has already made to its communications with 
producers as well as to changes to make the program simpler and 
more accessible, such as making program parameter changes, 
harmonizing its forms with the Canada Revenue Agency, 
reducing its supplementary form, and offering on-line My 
Account and calculator features. 


The Department agrees with the recommendation and is 
investigating the cost and feasibility of implementing additional 
accuracy determination methodologies to supplement 
approaches, which it will implement if they are found to be cost- 
effective and feasible. Added to the significant progress the 
Department has already made to reduce errors, implementing 
this recommendation could further enhance its current 
approach to internal reviews and verification. 


The Department agrees with establishing a target for the 
accuracy of payments and, as noted, has in fact established an 
internal target. The Department will also take into consideration 
the recommendation to report to Parliament, and other 
interested parties, on its achievements in meeting payment 


accuracy targets. 
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4.58 When estimating errors in 
payments from farm income support 
programs, Agriculture and Agri-Food 
Canada should include zero payment 
applications in its post-payment 


samples. (4.47—-4.57) 


4.60 When processing applications for 
benefits, under farm income support 
programs, the emphasis that 
Agriculture and Agri-Food Canada 
places on efforts to prevent, detect, and 
correct both under-payments and over- 
payments to producers should be 
proportional to the relative risks of 
underpayments compared 

to overpayments—as determined 
through the Department’s revised post- 
payment sampling results and as 
recommended elsewhere in 


this chapter. (4.32-4.59) 


474 Agriculture and Agri-Food 
Canada should strengthen its focus on 
continuous improvement in the 
administration of its farm income 
support programs by systematically 


* collecting relevant information about 
the nature, extent, and timing of 
errors; 


° analyzing the information; and 


* continually applying the results of the 
analyses to improve all phases of 
application processing, under the 
Department’s farm income support 


programs. (4.61—4.73) 


The Department agrees with the recommendation and is 
investigating the cost and feasibility of implementing additional 
accuracy determination methodologies, including sampling zero 
payment files, which it will implement if they are found to be 
feasible and cost-effective. 


The Department agrees with the recommendation. It will 
undertake, as indicated in its responses to other 
recommendations elsewhere in the chapter, to adjust its review 
and verification processes; to ensure a balance between 
overpayments and underpayments; and to ensure that, by 

the time payments reach producers, both have been identified 
and corrected to the same degree. Efforts in this regard will be 
proportional to the relative risks of underpayments compared 
with overpayments and will reflect both producers’ interests and 
the overall integrity of the program. 


The Department agrees with the recommendation to strengthen 
its focus on continuous improvement. Specifically, the 
Department agrees that using all the information it has available 
more systematically will ensure that it builds on past 
improvements, while fostering a stronger continuous 
improvement culture. 
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482 Agriculture and Agri-Food The Department agrees with the recommendation and will 


Canada should set realistic service time continue working with provincial partners to establish national 
standards for processing farm income service standards for the CAIS program to improve the 
support program applications and predictability of CAIS processing. 


should accurately measure its 
performance against those standards. 


(4.75-4.81) 


Provisions of the federal-provincial-territorial agreements 


490 Agriculture and Agri-Food The Department agrees with the need to meet the audit and 

Canada should ensure that it meets the _ reporting requirements of the federal-provincial-territorial 

audit and reporting requirements of the agreements. In this regard, it has already been working with 

federal-provincial-territorial provinces on finalizing audit parameters and carrying out audits 

agreements. (4.83-4.89) and will accelerate this work to ensure that audits are conducted 
in a more timely manner. 


Performance monitoring and reporting 


4.96 Agriculture and Agri-Food The Department agrees with this recommendation. It will, in 
Canada should conjunction with its provincial counterparts, develop targets for 
each of the measures in the Results-Based Management and 
Accountability Framework and report progress and 
achievements, on these targets, in the annual departmental 
performance report to Parliament. This task will be completed by 
December 2007 and will include, for example, targets related to 
producer participation in CAIS and to timeliness of payments. 


establish specific and measurable 
targets for all of the performance 
indicators in its Results-Based 
Management and Accountability 
Framework; and 


produce timely reports to Parliament 
on the performance of farm income 
support programs, measured against 


those targets. (4.91-4.95) 
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Conflict of interest 


4110 Agriculture and Agri-Food 
Canada should strengthen measures to 
reduce the risks of potential conflicts of 
interest. At a minimum, the 
Department should strengthen its 
values and ethics program—making it 
clear to employees that they cannot 
prepare applications for others for a 
fee—and establish processes to 


identify processors who also prepare 
applications for farm income support 
programs; 


address risks associated with 
employees preparing applications for 
their own farms or their families’ 
farms, including performing on-farm 
audits if necessary; and 


ensure that all applications prepared 
by farm income support program 
employees are approved manually 
rather than electronically. 


(4.97-4.109) 


The Department agrees. It has been strengthening, and will 
continue to strengthen, its values and ethics program, by 
continuing to advise all new staff of their obligations, under the 
Values and Ethics Code for Public Servants, and reminding them of 
their obligations on an annual basis. Recently, the Department 
strengthened its conflict of interest processes to proactively 
involve management in the resolution of potential or actual 
conflicts of interest. The Department has also recently instituted 
a formal training program and will be delivering this program 
across the country. Additionally, it has instituted a 1-800 hotline 
number where employees and managers can get immediate 
assistance. 


CAIS processors are required to self identify if they are 
completing a CAIS application for themselves, for a family 
member, or for friends. They are also asked to identify that 
application, and these applications are then processed by 
someone other than themselves in a separate team designated 
for that function, effectively distancing the employee from that 
application. Finally, the Department has informed employees 
through a Deputy Minister letter that they cannot prepare CAIS 
applications for a fee. This will be repeated on a yearly basis. 
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Chapter 


Managing the Delivery of Legal Services 
to Government 


Department of Justice Canada 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Progress since our 1993 audit 

The Department has made recent efforts to improve strategic planning 

The Department has improved its management of external legal agents in some areas 
Management improvements are required for alternative dispute resolution initiatives 
Reporting to Parliament needs improvements 


Quality assurance 


The Department has not articulated what it means by quality legal services 
The Department has elements of quality management 
The Department does not have systematic and reliable assurance of the quality of its services 


Resource management 


The Department does not use workload and human resources information 
to manage cost-effective legal services 


Financial arrangements with government departments continue to provide few reasons 
to improve resource management 


The administration of financial arrangements is inefficient 
Financial information provided to client departments is weak and inconsistent 


The Department needs better professional management 
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List of recommendations 
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Main Points 


Each department and agency of the federal government requires legal 
counsel to provide it with legal advice, to draft regulations and 
legislation, and to represent it in court. The Department of Justice 
Canada (Justice Canada) provides these services to all departments 
and agencies, with a few exceptions. In addition, it enforces regulations 
and legislation through prosecution, and examines the need for 
changes in legislation. We examined whether the Department 
effectively manages the delivery of legal services to meet the needs of 
government. We also looked at whether it takes appropriate steps to 
ensure the quality of the legal services and whether it delivers them in 
a cost-effective manner. 


Up to the end of our examination, prosecution services were provided 
by the Federal Prosecution Service—a branch of Justice Canada—and 
were included in our audit scope. Prosecution services became the 
responsibility of the Public Prosecution Service of Canada when it was 
created under the Federal Accountability Act, which received royal 
assent on 12 December 2006. This new organization is separate from 
Justice Canada and reports to the Attorney General. Our findings and 
recommendations relate to the work now carried out by the Public 
Prosecution Service of Canada as well as the work of Justice Canada. 


Canadians rely on lawyers to provide them with advice on, for 
example, purchasing or selling property, entering contracts, or settling 
disputes. Government departments and agencies rely on their lawyers 
for the same services. Justice Canada also covers a range of legal 
advisory, litigation (including prosecution), and legislative services. 
The Department’s job is complicated by the fact that its lawyers 
represent the interests not only of each client department but also of 


the government as a whole. 


Approximately 2,500 lawyers work for Justice Canada. Since our last 
audit in 1993 the annual cost of the Department's legal services has 
more than tripled to over $600 million. This significant increase is 
attributed to growing complexity and volume of litigation and 


prosecutions, as well as the growth in demand for legal services to 
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What we found 


departments since the introduction of the 1982 Charter of Rights and 
Freedoms. The Department has also faced more demand for its services 


in areas such as Aboriginal affairs, taxation, drug prosecutions, and 


immigration. 


The Department puts considerable effort into ensuring the quality of 
the legal services it provides to the government—its stated objective. 
While it does not have an overall quality management system, it uses 
several practices to promote quality. To date, its quality management 
efforts have focused on complex and high-risk litigation. It has also 
invested in information systems, development of guide books, and 
training and development of its staff. However, the Department has 
not expressed what it means by quality. This impedes its ability to 
deliver consistent quality of service to its 42 client departments and 
assess whether it is meeting its objective. It also lacks a system to 
provide senior management with ongoing assurance that all services 
meet established minimum standards. 


The Department lacks information on the volume of its work and its 
use of staff—information that would demonstrate whether it is 
delivering legal services cost-effectively. It has some time-keeping 
data for some lawyers, but recording time worked has only recently 
become a department-wide requirement. Justice Canada does not 
collect and use department-wide financial and workload information 
for monitoring trends, planning, setting priorities, managing 
resources, analyzing service delivery options, or identifying 
opportunities to control costs. 


Justice Canada shares the cost of legal services with departments. It 
has numerous financial arrangements with client departments— 
more than 100 arrangements with the eight departments we 
interviewed. Its financial arrangements are inconsistent, poorly 
documented, and inefficient to administer. Justice Canada has little 
incentive to control the costs of its services, because any 
unanticipated costs of legal services are borne by the client 
departments involved. Client departments also have little incentive 
to control their legal costs, which they are able to absorb as a small 
percentage of their total costs; for significant costs they can seek 
additional funding through the Treasury Board. Without good 
information on costs, it is difficult to manage and control the 
growing demand in departments for legal services and to find more 
efficient ways of providing them. 
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The total costs of legal services incurred across the government are 
not consolidated for reporting to Parliament. Justice Canada reports 
the total costs of its legal services and the total amount recovered 
from departments. Each client department might report the costs of 
the legal services provided to it by Justice Canada, but other legal 
costs paid by departments are not readily identifiable because they 
are not reported separately from other operating expenditures. 


Much of what we found had been identified in our 1993 audit as 
weaknesses. The Department has made progress in managing legal 
risk, managing legal agents, and introducing time-keeping into the 
Department. However, we found no evidence of results in some 
important areas, despite efforts by the Department—notably the use 
of department-wide information on workload and human resources 
and its financial arrangements with client departments. 


The Department has responded. The Department of Justice Canada 
agrees with each of our recommendations. Its detailed response follows 
each recommendation throughout the chapter. 
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Introduction 


The Role of the Department of Justice Canada 


5.1 The Department of Justice Canada (Justice Canada) can be 
characterized as Canada’s largest law firm with approximately 

2,500 lawyers and an estimated budget of close to $1 billion for the 
2006-07 fiscal year. This includes approximately $625 million for legal 
services provided to government. These services include providing 
legal advice on departments’ policies and programs, drafting new and 
amending existing legislation and regulations, and representing the 
government in court—whether in defence of civil litigation or to 
enforce Canada’s laws through federal prosecutions. The remainder of 
the $1 billion represents funds for Justice Canada policy development, 
as well as cost-shared and contribution programs, such as Aboriginal 
justice, youth justice, and legal aid. These programs and policy 
development are not within the scope of this audit (Exhibit 5.1). 


5.2. At the time of our audit, Justice Canada was responsible for 
¢ developing policies and laws, 
* providing legislative services to government, 
* providing legal advisory and litigation services to government, 
* providing prosecution services, and 


* delivering legal programs. 


The Minister of Justice is also the Attorney General of Canada. The two titles reflect the 
two different aspects of the position. 


The Minister of Justice is responsible for more than 45 statutes and areas of federal 
law, such as human rights, family and youth law, administrative law, Aboriginal justice, 
general public law, and private international law. The Minister introduces policies and 
programs that not only change the law, but are also intended to influence the way 
the law is applied; how the legal system functions; and how individuals, such as 
victims of crime, and groups, such as support services in family law, interact with 

the system. The Department enacts reform and provides advice on all reforms related 
to criminal law. 


The Attorney General advises federal departments and agencies on behalf of the 
Crown, and represents the Crown whenever legal actions are taken against a federal 
body. The Attorney General also oversees the prosecution of violations of federal law in 
all the provinces as well as the prosecution of all federal offences, including Criminal 
Code violations, in the territories. 


Source: Department of Justice Canada 
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Exhibit 5.1 Scope of the audit 


Departmental responsibility 


Providing legislative services 
to government 


Within the scope of this audit 


Examples of departmental responsibilities 


Drafting and reviewing legislation and regulations in 
both official languages, reflecting both the common 
and civil law traditions in Canada. 


Providing legal advice to 
government 


Providing legal advice on behalf of the Crown to 
government departments and agencies. The 
Department is mandated to provide legal advice only to 
the federal government. 


Litigation services to 
government 


io 
Representing the Crown whenever legal actions are 
taken against a federal body. 


1 


Providing prosecution 
services 


Departmental responsibility 


Development of Justice 
Canada policies and laws 


Undertaking federal prosecutions and criminal 
litigation on behalf of the Attorney General of Canada. 


Outside the scope of this audit 


Examples of departmental responsibilities 


The Minister and the Department are responsible for 

| 45 statutes and laws. The Minister has sole or shared 
responsibility to Parliament for these laws, including 
Canadian-Human Rights, Youth Criminal Justice, 
Access to Information, Divorce, and Official Languages. 


Programs and initiatives to 
support Justice Canada 
policies 


The Department develops and administers programs in 
support of the 45 statutes and laws in areas such as 
human rights, youth justice, Aboriginal justice, family 
justice, criminal law, privacy, and official languages. 


Source: Based on information provided by the Department of Justice Canada 


5.3 From its headquarters in Ottawa, Justice Canada delivers legal 


services to departments and agencies (referred to as departments 


throughout the chapter) and in six regions located across the country 


that primarily provide litigation and prosecution services. In 1995 the 


Department introduced a “portfolio” structure intended to deliver legal 
services to client departments more effectively. These portfolios include 
Business and Regulatory Law; Tax Law; Aboriginal Affairs; Citizenship, 
Immigration, and Public Safety; and Central Agencies. The Department 
also developed specialized centralized branches for civil litigation, federal 
prosecution and public law. While the regions report to one of three 
associate deputy ministers for operational purposes, they obtain their 
budgets and receive functional guidance from each of the portfolios and 
branches. As well, 42 legal services units located on the client 
departments’ premises deliver advisory services to client departments 
with oversight from headquarters. 
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5.4 Justice Canada told us that it faces many challenges. 

The complexity and volume of litigation has continued to increase, 

as has the demand for legislation and legal advice to departments since 
the 1982 Charter of Rights and Freedoms. Pressures are particularly 
significant in such areas as taxation, drug prosecutions, immigration, and 
Aboriginal issues. Unlike a private sector law firm, the Department does 
not have the option to choose the cases it will undertake. It must 
respond to incoming lawsuits in civil litigation and criminal prosecution 
from police agencies and, as a result, does not control the volume of 
incoming work. Consequently, there has been a significant increase in 
the costs of the legal services the Department delivers to government. 
We found that Justice Canada’s total operating expenses more than 


doubled from 1997-98 to 2004-05 (Exhibit 5.2). 


5.5 Canadians see the role Justice Canada plays almost every day 

in the news. The Department is involved in challenges to immigration 
and deportation decisions and rulings, class action suits against the 
government on lost revenue from Bovine Spongiform Encephalopathy 
(BSE or Mad Cow Disease) and tainted blood. Other cases involving 
Aboriginal rights, security certificates, and prosecution of individuals 
and groups involved in organized crime can include constitutional 
challenges. 


Exhibit 5.2 Justice Canada’s operating expenses more than doubled from 1997-98 to 200405 


Operating Expenses 
($ Million) 


600 
500 
400 
300 
200 


100 
1997-98 1998-99 1999-00 2000-01 2001-02 2002-03 2003-04 2004-05 


1. In 2003-04 the Department revised its reporting of expenditures to include approximately $117 million 
for lawyers providing services to other government departments, whose salaries had been reimbursed by 


the departments. 
2. Operating expenses included the Canadian Firearms Centre until 2003-04, when it was transferred to 
the Minister of Public Safety and Emergency Preparedness and renamed the Canada Firearms Centre 


Source: Public Works and Government Services Canada, Public Accounts of Canada 
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Progress since our 1993 audit 


5.6 Less visible are cases where Justice Canada lawyers represent the 
Crown, for example, in challenges to an individual’s income tax 
assessment or claims for compensation for injuries sustained when a 
dog bites a postal worker. On a daily basis Justice Canada also provides 
advice to every government department in developing government 
policy, legislation, and regulations, as well as administrative matters 
related to staffing and contracting. 


Focus of the audit 


5.7 In this audit we examined whether the Department was 
effectively managing the delivery of legal services to meet the needs 
of government. We also examined whether the Department has 
assurance it is delivering quality legal services, and whether it is 
delivering these services in a cost-effective manner. In the course 
of our audit, we determined whether the Department had made 
progress since our last audit in 1993. 


5.8 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


5.9 In 1993, the Office of the Auditor General conducted an audit of 
Justice Canada’s Legal Advisory and Litigation Services (Chapter 17). 
We found certain weaknesses in its management of the delivery of legal 
services. In this audit, we revisited some of these issues—specifically 
strategic planning, management of external legal agents, alternative 
dispute resolution initiatives, and reporting to Parliament. 


The Department has made recent efforts to improve strategic planning 


5.10 In 1993 we found the Department had neither a corporate plan 
nor equivalent plans at the sector level and was only beginning to 
implement processes for priority-setting and planning. 


5.11 At the time of this audit, we expected to see improvements in 
the Department’s corporate strategic planning processes. These 
processes provide a framework to assess and set priorities and to 
allocate resources required to deliver legal services. A strategic plan is 
also the basis for managing overall financial resources and coordinating 
the strategies and action plans for all Department sectors. 
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Standing agents—lawyers or law firms 
contracted to carry out prosecutions in areas 
where the Department does not have a regional 
office or where travel costs would be prohibitive. 
As well, the Department uses standing agents in 
larger cities for routine prosecution cases. 
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9.12 Since 1993 the Department has made efforts to implement better 
strategic planning and management practices. For example, the 
Department had developed a document—Strategic Plan 2001-2005. 
However, in this period, operational plans were not formally aligned with 
this corporate strategic plan. Without this alignment, there is a risk that 
the Department is not coordinated in achieving its corporate objectives. 
Further, we found that the former strategic plan had not been updated 
over the four-year period, rendering it out of date. At the time of this 
audit, the Department was in the process of establishing a new strategic 
plan as well as improving its operational planning and performance 
measurement systems. However, the Department has not completed 
implementing these improvements. 


5.13 Recommendation. Justice Canada should complete its corporate 
strategic planning and develop strategies to meet its long-term goals. 


The Department’s response. The Department agrees with the 
Auditor General’s recommendation. The Department created the 
Office of Strategic Planning and Performance Management in 
February 2005 to begin to strengthen the Department’s strategic 
planning and performance management capabilities, including 
integrating planning and strategic analysis, and performance 
measurement and reporting for the Department. The Department is 
also developing a corporate human resources strategy, which 
encompasses the key strategic human resources priorities of the 
Department as well as other human resources management initiatives 
in support of government-wide priorities. The Department will 
continue its efforts to implement planning frameworks and processes 
that reflect the unique needs and challenges that it faces in meeting its 
strategic outcomes. 


The Department has improved its management of external legal agents in some areas 


5.14. The Department hires two types of external legal agents for its 
litigation services. Standing agents provide prosecution services for 
indeterminate periods, while civil agents work on a case-by-case basis. 
Note that Justice Canada is not required to follow government 
procurement policies when appointing legal agents. The Minister of 
Justice is responsible for all appointments of external legal agents both 
for Justice Canada and client departments and has primary 


responsibility and accountability for agents’ performance. 


5.15 Standing agents. In our 1993 audit we reported problems where 
standing agents of the Federal Prosecution Service (FPS) received 


little supervision. The Department recognized the need for 
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Civil agents (or ad-hoc agents) —lawyers or 
law firms contracted to carry out civil litigation or 
other civil work on behalf of the Department due 
to their specific area of expertise or the lack of 
available Justice Canada lawyers. 


improvements in its information on what agents are doing and how 
they are performing. We expected to find that the Department had 
corrected these deficiencies and that it effectively manages its use of 
standing agents. 


9.16 In 2000 the Federal Prosecution Service (FPS) implemented an 
Agent Affairs Program to better manage its standing agents. A 
subsequent internal audit in 2003 found that certain procurement 
problems continued to exist. In our current audit we found that 

since 1993 there have been improvements to the procurement and 
management of agents hired for federal prosecution. These 
improvements include better processes to identify agents whose 
charges exceed expected amounts, and to take follow-up action. 


9.17 We selected 25 FPS standing agent files. In our review we found 


* evidence that screening processes were used for those counsel 
who were recently engaged, 


* documentation on the ongoing communication and supervision of 
the agents, and 


* reviews for reasonableness of invoices submitted by standing 
agents were performed and signed by the supervision unit. 


However, in the file review we found no formal evaluations of the 
standing agents and no documentation of corrective action taken 
where poor performance had been noted. 


9.18 Civil agents. In our review of 21 contracts for legal agents for 
civil litigation and civil work, we examined the processes for the 
selection, appointment, and management of these agents. We found 


* no documentation of an in-house search for qualified counsel 
prior to seeking outside counsel; 


* no documented rationale for the selection of the legal agent; 


* no consistent basic information, such as start and end dates, 
estimated number of hours, estimated total value of the work, and 
terms of work in the agreement between the outside lawyer and 
the Department; 


* no consistent monitoring of the ongoing costs of an agreement. In 
some agreements, numerous detailed invoices received were not 
summarized and added together to inform the Department of the 
ongoing cost of the case; and 


* no documented evaluation of the performance of civil litigation 
agents was required for their files, according to the Department’s 
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policy for civil litigation agents—whether for good or poor 
performance. Only poor performance would be brought to the 
attention of a supervisor. It was not clear how performance 
information on an agent would be used in the negotiation or 
selection of agents for future work. 


5.19 While we are aware that the Department is taking a number of 
steps to improve the processes for hiring and managing civil agents— 
such as establishing a pool of eligible law firms for civil agent work—it 
is too early to assess the results. 


5.20 Recommendation. Justice Canada should examine the 
weaknesses noted in its management of legal agents used for civil work 
and take corrective action. 


The Department’s response. The Department agrees with the 
Auditor General’s recommendation. In fact, the Department has an 
action plan to support the continued implementation of improvements 
to the management of legal agents. This action plan will also address 
the issues raised by the Auditor General. 


Management improvements are required for alternative dispute resolution 
initiatives 

§.21 In our 1993 audit we noted that the Treasury Board approved 
recommendations arising from the first phase of the joint review of the 


justice and legal system and recommended the Department review 
alternative dispute resolution mechanisms as a means to reduce costs. 


5.22. In this audit we found that Justice Canada, in conjunction with 
client departments, has implemented a number of alternate strategies 
to litigation over the years in order to avoid costly court processes. We 


assessed five strategies: 
¢ the Competition Bureau's use of Alternate Case Resolution, 


¢ Fisheries and Oceans Canada’s project for Controlling 


Prosecution Costs, 


* Indian and Northern Affairs Canada’s Saskatchewan Treaty Land 


Entitlement Settlement program, 

¢ Justice Canada’s pilot project in its Prairie Region to promote 
mediation in Alberta for resolution of litigation files, and 

* the establishment of the Canadian Food Inspection Agency’s 


centre of expertise on Dispute Resolution to avoid litigation by 


using alternate dispute resolution. 
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5.23 We expected each of the initiatives to have clear objectives, 
scope, reasonable timelines, adequate resources, and ongoing 
monitoring and review mechanisms in place, and that the initiatives 
were implemented within the timelines established and within budget. 


5.24 We found that most of the documentation on the initiatives 
lacked clear objectives and estimates of potential cost savings. We did 
not find an identified project management structure that would guide 
the initiatives, and there was no formal review of outcomes against the 
original objectives. 


5.25 Recommendation. In addition to its current management of 
alternative dispute resolution initiatives, Justice Canada should ensure 
that it clearly defines each initiative’s objectives, estimates the 
potential cost savings and formally reviews the outcome of initiatives 
against their objectives. 


The Department’s response. The Department agrees with the 
Auditor General’s recommendation, which is consistent with the 
recent review of the Department's dispute resolution initiative that has 
resulted in a new mandate focused on providing services to 
departments supporting long-term prevention and early resolution of 
disputes. These client-focused services include systems design, legal 
advisory support, and training, all of which are linked to the 
departmental priorities of improving practice management and 
managing the volume of litigation. 


Reporting to Parliament needs improvements 


5.26 In our 1993 audit, we noted that Parliament was not aware of the 
full cost of delivering litigation and advisory services because client 
departments bore significant proportions of the costs. 


5.27. In our current audit we found there is still no reporting to 
Parliament on the overall cost incurred across government for legal 
services. Justice Canada reports the total costs of its legal services and 
the total amount recovered from departments. Each client department 
might report the costs of the legal services provided to it by Justice 
Canada, but other legal costs paid by departments are not readily 
identifiable because they are not reported separately from other 
operating expenditures. For example, we reviewed a study of estimated 
costs for the 2003—04 fiscal year prepared by the Justice Canada legal 
services unit for Transport Canada, which is a large user of legal 
services, and found that Justice Canada paid $7.8 million in legal 
services provided to Transport, of which $5.2 million was cost 
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recovered. However, Transport Canada incurred an additional 

$1.3 million for legal services support and legal agents. Some of these 
additional costs are not easily identifiable as legal costs because they 
are included in Transport Canada’s reports to Parliament as operating 
expenditures. There is no consolidated report to Parliament on the 
total costs incurred by government for legal services. We were not able 
to estimate this amount as the data is not available from Justice 


Canada. 


5.28 We also reviewed the Department’s annual reports to 
Parliament: the Report on Plans and Priorities and the Departmental 
Performance Report. In our assessment of the performance reporting, 
we found that the Department clearly outlined its main objectives and 
provided financial information on the costs of these objectives. 
However, there were few meaningful performance measures in its 
reports to Parliament. As well, the information provided in the annual 
performance report was not aligned with the plans and priorities for 
the period. Justice Canada is undertaking an initiative to improve its 
performance reporting. 


5.29 Recommendation. Justice Canada should define performance 
measures for its corporate priorities and improve its performance 
reporting to Parliament. 


The Department’s response. The Department agrees with the 
Auditor General’s recommendation. As part of its departmental 
performance report to Parliament, the Department will incorporate 
relevant measures from the annual assessment of our Management 
Accountability Framework completed by the Treasury Board 
Secretariat. 


More broadly, the Department has implemented a number of changes 
to its annual reports on plans and priorities and departmental 
performance reports in an effort to improve reporting to Parliament 
and the public on the results of departmental initiatives and priorities. 
Also, as part of its efforts to improve reporting, the Department has 
implemented a practice whereby a committee of performance 
measurement specialists within the Department, as well as one 
external adviser, assess the quality of data used to arrive at 
management's assessments of the degree to which the Department is 
achieving its priorities each year. The results of these reviews are 


reported in the departmental performance report. 
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Quality assurance 


5.30 Central to Justice Canada’s mission is the delivery of high-quality 
legal services to government. Quality assurance is a systematic process 
that should provide senior management with ongoing and reliable 
assurance that all services meet minimum quality standards. 


5.31 In particular, we expected the Department to 


¢ have articulated an expression of quality and communicated it 
across Justice Canada and with client departments in order to 
promote a consistent understanding and expectation of quality 
services; 


¢ manage the quality of its legal services, by assessing the level of 
risk and assigning appropriately qualified staff who deliver 
relevant legal services in a timely and reliable manner; and 


¢ have a system in place to provide senior management with 
ongoing and reliable assurance that all services meet established 
minimum standards. 


5.32 During our audit we found few areas where the Department 
was doing well and more areas where it could make improvements. 
The following observations provide details on our findings. At the end 
of this section, we provide one recommendation that addresses the 
need for a comprehensive quality assurance system. 


The Department has not articulated what it means by quality legal services 


5.33 We found that the department has not articulated what it means 
by quality legal services. Most lawyers we asked had their own 
definition of quality, for example, performing thorough research on the 
legal issue, providing advice that is consistent with previous positions 
taken, and writing clearly, to name a few. However, without a 
departmental expression of quality, it is difficult to have a consistent 
understanding of quality legal services across the Department. Further, 
this lack of an expression of quality makes it difficult for the 
Department to assess whether it is delivering legal services that meet 
its quality expectations. 


5.34 Justice Canada delivers legal services to 42 client departments 
across the government. We expected that Justice Canada would reach 
an agreement with each client department on the quality of legal 
services to be provided. We did not find an agreed-upon, written 
expression of quality legal services used consistently for all client 
departments. Without this shared understanding, it is difficult for Justice 
Canada to meet client department’s expectations and to determine 
whether it is delivering quality consistently across the government. 
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9.35 Justice Canada needs to clearly articulate what it means by 
quality legal services and ensure that this is understood and used across 
the Department and with all client departments. 


The Department has elements of quality management 


5.36 We found that the Department does not have a formal, 
systematic approach to managing quality; however, it provided us with 
a list of its quality management practices: 


* legal risk management; 
* staff training and development; 
* tools for lawyers such as information systems and guide books; 


* service standards defined for a number of portfolios, branches, and 
regions; and 


* practices to deliver services to client departments. 


In the absence of an overall framework, we examined these practices 
and present the following observations. 


5.37 Legal risk management. Since 2000 the Department has sought 
to manage legal risks more effectively, starting with a focus on litigation 
services. While the Department is not yet consistently using legal risk 
assessment in its advisory work, we were informed that it is developing 
new principles and practices specific to advisory services. Our audit 
focused on the systems in place to manage litigation risk. 


5.38 The Department uses litigation risk management to guide the 
ongoing monitoring and reporting of civil litigation and prosecution 
cases. Justice Canada lawyers assess the level and impact of risk to the 
government's policies, laws, programs, or finances; to federal/ 
provincial/territorial or international relations; and to public 
confidence in government or the courts. They evaluate the likelihood 
and assess 


of an adverse outcome of a case—low, medium, or high 
the potential impact—minor, moderate, or significant (Exhibit 5.3). 
The Department uses a three-by-three grid and standard criteria to 
assess risks and develop management actions. This risk assessment is 
prepared by the lead counsel on the case who is expected to change the 


assessment if necessary as the case proceeds. 


5.39 We found the litigation risk management framework to be a good 
tool that provides an effective means of analyzing new cases and 
communicating their risk in a consistent fashion. This practice is 
designed to share the understanding of the risks and rankings of cases 
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Exhibit 5.3 The Department has a good risk assessment tool that incorporates management actions 


Risk Level (Risk Management Actions) 


Considerable | Must manage and | Extensive = 
Significant | management | monitor risks =| management 
required Me | essential. a 


Risk Level 7 “RiskLevel8 Risk Level9 


Risks may be worth | Management effort Management effort 
Moderate | accepting, with worthwhile required. 5 Q oS 
monitoring Pearce secant 


Impact 


Risk Level 4 Risk Level 5 | Risk Level 6 


Accept risks Accept, but monitor | Manage and 


Minor : , : 
risks monitor risks 


Risk Level 1 Risk Level 2 Risk Level 3 


Low Medium High 


Likelihood of adverse outcome 


Source: Department of Justice Canada 


across the Department and to support the management and mitigation 
of the risks. As well, resources are assigned depending on the risk level 
and complexity of the case. 


5.40 At the Department, files with higher risk levels receive more 
attention from senior management. Cases assessed as having a 
potentially significant impact (risk levels 7 to 9) are supposed to be 
reported to the Deputy Minister on a regular basis, while cases assessed 
from | to 6 are not likely to be discussed beyond the regional level. 
Central committees discuss key litigation cases and class action 
litigation in order to develop broad litigation strategies, resolve 
differing views on legal issues, and ensure consistency in arguments. 
In our review of 91 files, we found evidence that portfolio heads and 
committees reviewed and approved strategies and arguments for 
high-risk litigation cases, as required by the Department. 


§.41 While the litigation risk management framework is a good tool, 
its effective use depends on the correct initial assessment made by 
individual lawyers. We found that, in some instances, the lead counsel’s 
risk assessment was not challenged. This could lead to inadequate 
preparation for the case. While we have some evidence that Justice 
Canada and client departments did joint risk assessments, we found no 
documented evidence of this in the case files we reviewed. Justice 
Canada and client departments need to review and approve the initial 
risk assessment score and any subsequent need for changes to the score. 
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5.42 The current litigation risk model assigns senior management 
attention to a case assessed as a 7, which has a low probability of an 
adverse outcome. Meanwhile, a case assessed as a 6, which has a high 
likelihood of an adverse outcome with a moderate impact, does not 
receive senior management attention, but receives attention in the 
region. We consider the level 6 designation may not ensure the 
appropriate management action to mitigate risk. The Department is 
aware of this issue and has decided to continue with this grid. However, 
it needs to ensure that special attention is given to cases where a 
potentially incorrect score could have a considerable impact on its 
oversight of the case. 


9.43 As noted earlier, the Department does not consistently use risk 
management in its advisory services at this time. As a result, it is not in 
a position to properly assess these legal risks and to mitigate them. 
Furthermore, Justice Canada does not have the risk assessment 
information that would help allocate the appropriate quantity and 
quality of legal resources to advisory work. 


5.44 Human resources management. Planning and managing 
human resources are important in the delivery of quality legal services. 
However, we did not find a human resources strategy in place at Justice 
Canada to ensure it can meet its goals, including ensuring the quality 
of staff. The Department did inform us that it has an initiative under 
way to identify its human resource management priorities based on 
operational business plans and government priorities. Still, we did not 
find a corporate retention strategy or a succession plan. Without a 
human resource strategy and related plans, the Department risks 
making staffing decisions that are inconsistent with its long-term goals, 
as well as not having appropriately qualified staff who deliver timely, 
reliable, and relevant legal services. 


§.45 Client departments rely on the quality and professionalism of 
lawyers assigned to their department. Maintaining and further 
developing this expertise is important. Justice Canada’s training and 
development policy includes five days of professional development per 
year per employee. However, we found that the Department did not 
consistently provide funds for professional development to support this 
policy. We also found that the Department does not consistently 
record training days taken, which limits the information available to 


monitor training and development. 


5.46 The Department has some programs for recruiting articling 
students and has a performance assessment program that identifies 


objectives and training needs, which provides feedback to individual 
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lawyers. It has also made efforts over the past 10 years to integrate 
defined competencies for lawyers and managers (skills, behaviours, and 
capabilities) into the human resource management system. However, a 
list of expected competencies for lawyers and managers was not in 
place at the time of our audit. 


5.47 The Department needs to have a human resource strategy and 
practices in place to ensure it has appropriately qualified staff to meet 
its needs for quality. These practices need to include a retention 
strategy and succession plan, as well as defined competencies for 
lawyers and managers. 


5.48 Quality and professionalism of lawyers. The Department relies 
heavily on the quality of the individual lawyers for all of its services. 
Lawyers specialize in certain types of law or services such as litigation, 
prosecution, legal drafting, or advisory services. They bring to their 
work the education, skills, expertise, and judgment of their profession. 
As lawyers gain experience with a client department, they develop 
knowledge of the department’s legislation, regulations, policies, and 
programs, and are better able to contribute to resolving the legal issues 
it faces. We found that the client departments we interviewed were 
satisfied with the quality of individual lawyers assigned to them, and 
were quite satisfied with the overall services provided. As well, the 
Justice Canada counsel who heads the legal services unit routinely sits 
as a full member on departmental executive committees. Client 
departments informed us that Justice Canada lawyers have improved 
their advice by identifying possible solutions to legal issues. 


5.49 Guidance, information, and documentation. To help ensure 
the quality and consistency of legal services, the Department has 
developed extensive guide books for both civil litigation and federal 
prosecution lawyers. All lawyers must follow the policies, processes, 
and templates provided. However, we found that the Department does 
not periodically assess compliance with these guides. Without periodic 
assessment, management will not know if the guides are effective and 
followed consistently. 


5.50 The Department also uses a number of information systems to 
help lawyers provide consistent litigation and advisory services. These 
systems provide Justice Canada lawyers with desktop access to 
electronic information such as legal precedents and previous key policy 
advice. They help manage electronic evidence as well as a large 
number of documents, and help ensure the quality advisory services by 
making legal opinions available to lawyers situated across the country. 


Chapter 5 


Report of the Auditor General of Canada—May 2007 


MANAGING THE DELIVERY OF LEGAL SERVICES TO GOVERNMENT—DEPARTMENT OF JUSTICE CANADA 


5.51 Unfortunately, lawyers housed in client departments find 

these information systems difficult to access. As well, we found that 
the Justice Canada database of legal opinions was not up to date. 
Without this, lawyers do not have the most recent information 
available, thus increasing the risk they may provide inconsistent 
advice. The Department has recognized that these systems are 
important to deliver quality legal services and that they need 
improvement. It recently approved a knowledge management strategy. 
However, full implementation of improvements to the systems will take 
years to accomplish. In the interim, the Department needs to improve 
the accessibility of the information systems and establish procedures to 
ensure the databases are up to date. 


5.52 In our review of files, we found a lack of documentation regarding 
key decisions. We expected documented legal advice supporting 
decisions to be in the files since it represents communication with 
client departments. We also expected this documentation to be readily 
available should there be a change in lawyers during the course of a 
trial or hearings. The Department needs to review and improve its 
documentation practices. 


5.53 Consistent legal decisions and advice. Given the decentralized 
delivery of legal services to all departments, we expected Justice Canada 
to have established mechanisms to provide consistent legal decisions 
and advice. The Department established a portfolio structure, which 
allows for numerous points of review to promote consistency when 
providing advice and opinions and drafting legislation. This structure 
includes extensive communication and peer review. 


5.54 We found that the portfolio structure requires a complicated flow 
of communications within Justice Canada and with its client 
departments for both litigation and advisory services. Exhibit 5.4 
provides an example of the communications required for a high-impact 
litigation case with Fisheries and Oceans Canada. 


5.55 We find that this structure, while providing multiple points of 
review, involves significant overhead (cost and time) for Justice Canada. 
While the Department believes that the structure is appropriate, it has 
not assessed whether its costs are reasonable. The Department needs to 


examine the appropriateness of the cost of this structure. 


5.56 In the majority of instances, the Department’s practices to 
ensure consistency appear to function well. However, in our interviews 
with departmental officials regarding advisory services, we found 


isolated instances where clients received more than one opinion on the 
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same issue from different Justice Canada lawyers. The departmental 
officials had expected and wanted a single national opinion from 
Justice Canada. We were also told of one instance where the differing 
positions were played out in front of the client who then had to act as 
conciliator. 


5.57 Service standards. Service standards are another tool Justice 
Canada has to ensure consistency in the delivery of quality legal 
services to client departments. The service standards we found 
addressed areas such as responsiveness, clarity of legal advice, and use 
of plain language. Establishing service standards has been a 
work-in-progress at the Department since our 1993 audit. At the time 
of this audit, we found that not all of the portfolios, regions, and 
branches had service standards in place. Some standards were drafted 
with client departments, but had not been implemented. 
Consequently, the practice of using service standards in discussion with 
clients is not consistent across the Department. Justice Canada needs 
to complete its service standards and use them consistently in 
discussions with all client departments. 


5.58 Formal feedback. Senior managers at Justice Canada are in 
touch with government departments on a regular basis in order to 
receive feedback on their services. Of the eight client departments we 
reviewed, portfolio heads at Justice Canada meet at least once a year 


Exhibit 5.4 Communications within Justice Canada and with its client departments are complex for 
high-impact cases 


Fisheries and Oceans Canada (DFO) Justice Canada 


Deputy Minister Deputy Minister 
ba ee 
Legal Risk Management National Litigation 
Committee Committee 


Justice 


Regional Office 
(British Columbia) 


Business & 
Regulatory Portfolio 


Regional Litigation 
Committee 


Justice 
Regional Legal 
Services Unit for 


DFO 
(British Columbia) 


Legal Services Unit 
for DFO 


Regional Office 
(British Columbia) Le (Ottawa) 
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with departmental senior officials to discuss the performance of the 
legal services unit. Managers of the legal services units also seek 
ongoing informal feedback throughout the year. We were told that 
client departments would not hesitate to bring quality issues to the 
attention of senior management at Justice Canada. 


9.99 The Department has also used client satisfaction surveys to 
receive formal feedback on its service. However, we found that the 
survey results were not comparable or done regularly. This does not 
allow the Department to compare the feedback across government or 
track changes in satisfaction over time. The Department recognized 
that there were problems with its survey methodology and is currently 
taking steps to improve surveys in order to have a government-wide 
perspective. It needs to complete these improvements in a timely 
manner. 


The Department does not have systematic and reliable assurance of the quality of 
its services 


5.60 We expected to find that Justice Canada would have a system in 
place to provide senior management with ongoing and reliable 
assurance that all products and services meet established minimum 
standards. This system would include the periodic assessment of the 
appropriateness of the design and whether the quality management 
framework provides it with assurance that it is delivering quality legal 
services. This assessment would help to identify weaknesses, allowing 
the organization to take corrective action. 


9.61 We found, however, that the Department is not assessing 
whether it has the appropriate design in place and whether it is 
functioning as intended. The weaknesses in the practices that we have 
noted in earlier paragraphs could have been identified through such an 
assessment. The Department does not have assurance that these 
practices are appropriate and effective in delivering quality in its legal 
services. Without measuring and assessing its quality practices, the 
Department is unable to manage quality. 


§.62 Recommendation. Justice Canada should develop a quality 
management system or systems for each of its practices to provide 
senior management with ongoing and reliable assurance that all 


services meet established minimum quality standards. 


The Department’s response. The Department agrees with the 
Auditor General’s recommendation and will build on the elements of 
its quality work recognized by the Office of the Auditor General with a 


view to having a sound quality management approach. 
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Resource management 


5.63 In light of the significant increases in legal services costs at 
Justice Canada since our 1993 audit, we expected the Department to 
have systems and practices in place to effectively manage its 
resources—that is, to know its costs and workload and to manage the 
delivery of legal services in a cost-effective and economical way. 


The Department does not use workload and human resources information 
to manage cost-effective legal services 


5.64 In our 1993 audit, we noted a lack of management information 
on workload, costs, and performance, which had an impact on the 
Department’s ability to manage its workload. 


5.65 In this audit we expected the Department to be using 
management information on workload volumes and costs for legal 
advisory services and litigation in order to monitor trends, plan, set 
priorities, manage resources, analyze service delivery options, and 
identify opportunities for controlling costs. 


5.66 We found that the Department has taken some steps since 1993 
to improve its collection of detailed timekeeping by lawyers for all types 
of legal services. The Department informed us that it introduced 
timekeeping in 1996. By 2003, timekeeping was used by most counsel, 
although not consistently. In December 2006, the Department made 
timekeeping mandatory for all counsel and paralegals that provide legal 
services to government departments regardless of the type of work 
performed—advisory, litigation, or legal drafting. Since consistent 
timekeeping across the Department was relatively new at the time 

of our audit, the Department did not have complete information to 
analyze and compare workload volumes and costs for all of its services. 


5.67 Without proper department-wide timekeeping data, the 
Department lacks information on work volume and its use of staff, and 
as a result, it has difficulty understanding and controlling its costs. 
Department managers need timekeeping and human resource 
information, not only to manage their workloads in the short term, but 
also to plan and prepare for the future. 


5.68 As with timekeeping information, we found that the department 
does not use its human resource information for department-wide 
analysis. While we found detailed human resource and cost 
information, the Department was not using it to estimate the future 
number and types of positions it will need or its future salary costs. Nor 
did we find departmental reports that tracked key human resource 
statistics such as vacancy and turnover rates. This lack of summary 
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level information limits the Department’s ability to analyze its 
resources, detect trends as they emerge, and make informed decisions. 


5.69 For example, we found that the Department had difficulty 
providing us with department-wide reports. It could not provide a 
summary report on historical salary expenses and the number of 
positions by staffing level with information from both its financial and 
human resources information systems, since these systems have not 
been reconciled. Also, some employee information available prior to 
the 2002-03 fiscal year is inconsistent with data collected after that 
year, making it difficult to analyze trends over time. 


9.70 We found the Department manages its finances in a decentralized 
manner and at a transactional level. Managers in portfolios, branches, 
and regions know what they have spent and can compare this against 
their budget. In addition, within the past year, the Department has 
begun to review a monthly overview of its expenses against its budget. 


5.71 In our review of 96 files as well as eight legal services units in key 
client departments, we found little evidence of Justice Canada using 
financial and workload information to monitor trends, plan, set 
priorities, manage resources, analyze service delivery options, and 
identify opportunities for controlling costs. 


9.72 Without key departmental work volume data, and financial and 
human resource management information that is collected, analyzed, 
and reported to senior management on a regular basis, the Department 
does not know whether it is delivering legal services cost-effectively. 


Financial arrangements with government departments continue to provide few 
reasons to improve resource management 


5.73 From the Glassco Commission in 1962 to the late 1980s, 
Justice Canada paid for almost all legal services for the entire 
government with the exception of the payment of judgments and 
settlement of claims. As the demand for legal services exceeded the 
Department’s resources, it sought to recover costs from its clients 
and adopted different financial arrangements with each department. 
The departments also directly pay for certain other legal costs, such 


as paralegals and administrative support staff. 


5.74 The government’s Common Services Policy requires that all 
departments across government use Justice Canada for its legal 
services. This policy prevents these departments from seeking other 


sources of legal services that could be more cost-effective but could 
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place the Crown at risk. The policy does require, among other things, 
that Justice Canada provide services in a cost-effective manner. 


5.75 In response to our 1993 audit where we raised a number of issues 
related to the financial arrangements with client departments, the 
Department agreed to undertake a study to identify the positive and 
negative features of current arrangements and options for them. We 
expected the Department to have analyzed alternatives to its financial 
arrangements with client departments, developed a clear rationale for 
using cost recovery, and to have implemented an appropriate system. 


5.76 During our current audit we found that the Department and the 
Treasury Board Secretariat had performed a number of studies on cost 
recovery and client service since the early 1990s. By the mid-1990s the 
Department had developed, through consultation with its client 
departments, a legal services policy framework. This framework was 
based on principles of joint planning; shared responsibility for costs; 
and shared accountability for performance, allocation, and use of 
resources. The Department considered this framework to be a means 
to manage day-to-day services more effectively, thus reducing the 
overall costs of legal services to government. The assumption was that 
departments would moderate or control their demand if they were 
paying part of the bill. 


5.77 A 2003 internal Justice Canada report stated that there 
continued to be few, if any, incentives to both manage and control the 
increasing demand for legal services, or identify more efficient ways to 
deliver legal services. In addition, the report noted that dedicating the 
same amount of funds as the previous year to each client department 
limits Justice Canada from reallocating funds from one department to 
another that may have a greater need for legal services. This report 
then outlined alternatives to the current financial arrangements that 
would provide a more consistent approach and better control over 
increasing demand and costs. The executive agreed to proceed with 
the proposed changes; however, we found no evidence that these 
changes were implemented. 


5.78 In 2004, the Department continued to address this issue by 
initiating a Review of Legal Services. The objectives of this initiative 
were two-fold: 


* to improve the sustainability of legal services delivery through 
improved strategies and a sustainable funding regime; and 


* to identify approaches to cost-effectively manage litigation. 
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§.79 At the time of our audit, this initiative was still under way, 
although most short-term efforts were focused on the first objective of 
sustainable funding for legal services. While the Department has 
developed a number of proposals for changes to the financial 
arrangements and continues its efforts and discussions, we found no 
tangible progress on the management and control of the demand and 
cost for legal services. 


5.80 We found that, in the process of developing these proposals, the 
Department had done little analysis to assess the assumption that cost 
sharing and recovery would control demand and reduce costs. In fact, 
since legal costs represent less than one percent of client departments’ 
total budgets, the additional costs are often not a significant factor in 
deciding whether or not to obtain additional services. Also, if 
unanticipated legal costs arise during the year for significant issues, 
client departments may seek additional funds through the Treasury 
Board. Thus, the current financial arrangements will continue to lack 
incentives to control costs. Since our last audit, Justice Canada has not 
been able to find a way to work with client departments to manage 
demand and thereby reduce costs. 


The administration of financial arrangements is inefficient 


§.81_ At the time of our audit, we were informed that there were 
multiple financial arrangements, with Justice Canada recovering up 
to 100 percent for its services. For the 8 client departments we 
reviewed, we found more than 100 financial arrangements. When we 
examined the administration related to these financial arrangements, 
we found it poorly organized and controlled, inconsistent, and with 
incomplete documentation. We found that the Department’s financial 
services area does not maintain a central collection of all the 
agreements between Justice Canada and client departments; instead, 
they are held in each portfolio. By the end of our fieldwork, we were 
not confident that we had received all of the agreements. 


5.82 Justice Canada has estimated a total cost of more than $2 million 
per year for all Department staff involved in running this overly 
complicated system. This estimate does not include the administration 


costs incurred by each client department. 


Financial information provided to client departments is weak and inconsistent 


5.83 Both the Department and its clients are stewards of the 
resources under their control. Effective stewardship requires 
information to effectively plan and manage demand for legal 


services and the use of resources. 
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5.84 We expected Justice Canada to be able to inform and report to 
client departments on the services provided and their overall cost and 
to have a written agreement in place. We believe that, as the service 
provider, Justice Canada is responsible for providing this information. 
In addition, we expected that Justice Canada would routinely provide 
client departments with the estimated costs of individual cases to 
improve client department’s decision making regarding the cases. 


5.85 In our review of eight client departments, we found a variety of 
financial planning practices in place. Three departments had good 
joint department-Justice Canada work plans, supported by detailed 
resource estimates, as well as periodic tracking against these plans. In 
others, no documentation of consultations was available. 


5.86 In our interviews with client departments, most managers using 
the services had little understanding of the overall cost of legal services 
provided to their department. While the client departments approve 
each individual detailed invoice, Justice Canada could not easily 
provide a roll-up of the financial information on the overall costs of 
legal services to each client department. In a few instances, we found 
that client departments tried to develop their own reports in order to 
better understand and predict the overall costs of their legal services. 


5.87 On acase-by-case basis, when a client department requests an 
opinion or needs legal services, Justice Canada discusses the client’s 
priorities and deadlines. However, it does not consistently provide cost 
information to allow the client department to make informed decisions 
about the legal services it requests. 


5.88 We reviewed 53 cases that proceeded to litigation or cases that 
were appealed. While most of the Justice Canada files contained 
documentation of its advice to clients on whether to proceed, we did 
not find documentation on the estimated costs. We did, however, find 
a forecast of expenditures for the coming year for certain significant 
cases. Still, the client departments did not receive cost information 
from Justice Canada to help them make an informed decision. 


The Department needs better professional management 


5.89 During our audit we found that the Department has continued 
to make efforts, over the past 10 years, to develop and implement a 
number of initiatives to improve administrative practices. We noted 
many of these earlier in the chapter, including strategic planning, legal 
risk management, information systems for lawyers, formal client 
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surveys, timekeeping, cost recovery through a legal services policy 
framework, financial and human resource information and reporting, 
and the Review of Legal Services project. 


9.90 We found that the implementation of these initiatives had been 
ongoing for many years and, at the time of our audit, were still not fully 
implemented. As a result, we found inconsistent administrative 
practices across the Department. In our view, Justice Canada does not 
have the tools in place to provide management with the information it 
needs to support departmental decision-making. 


5.91 The responsibility for management lies with the Deputy Minister. 
However, given the size of the Department, we expected to find a 
senior manager with clear delegated authority for the management of 
administration. This position would complement the Deputy 
Minister’s responsibilities for the law. The Assistant Deputy Minister 
(ADM) for Corporate Services co-chairs three committees—finance, 
human resources and information technology—that make 
recommendations to the main decision-making body—the Senior 
Management Board. The ADM is also a member of the Senior 
Management Board. However, this ADM position does not have the 
authority to make significant decisions for these areas. 


5.92 In our opinion, the lack of a senior executive position with 
delegated authority for management practices has limited the 
Department's ability to focus on and implement the improvements 
necessary to manage a large department that is responsible for 
delivering cost-effective professional services. 


5.93 Recommendation. Justice Canada should make improvements 
to its resource management. The Department should 


* use management information on workload volumes and costs for 
all of its legal services to better monitor trends, plan, set priorities, 
manage resources, analyze service delivery options, and identify 


opportunities for controlling costs; 


* work with client departments to effectively plan, manage, and 
improve the cost-effectiveness of their financial arrangements 
with a view to managing demand for legal services and thereby 


reducing costs; and 


* ensure that a senior manager with the necessary authority leads 
the improvement of management practices at the Department 


and oversees implementation of these changes. 
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The Department’s response. The Department agrees with the 
Auditor General’s recommendation. The Department of Justice 
Canada will continue the work underway to develop approaches to 
further develop its capacity to use management information to 
enhance management decisions, and regularize its cost recovery 
agreements with government departments. 


The Department is currently working on putting in place an initiative 
to better coordinate and integrate its law management practices. 

The Department will, at the same time, review its management 
practices to identify and put in place corporate reports to provide 
management information to support departmental decision making. 
Senior management authority to lead our work on management 
practices will be strengthened. 


Conclusion 


5.94 Justice Canada has made progress since our 1993 audit, mainly 
in the area of managing external legal agents for prosecutions. There 
remains, however, the need for improvements in managing legal agents 
for civil work, strategic planning, and managing important initiatives. 
The Department also needs to improve its performance reporting 

to Parliament. 


5.95 We found that the Department does not have the necessary 
system or systems to provide assurance that it is delivering quality legal 
services to government; but it does have some elements of quality 
management that it can build on. The Department has not assessed 
whether it had the appropriate design in place and whether it was 
functioning as intended. We found some practices in place to manage 
quality such as litigation risk management. However, we also identified 
weaknesses, which could have been identified through such an 
assessment. The Department needs to develop a quality management 
system or systems for each of its practices to ensure that quality is built 
into its services and to provide senior management with ongoing and 
reliable assurance that all services meet established minimum quality 
standards. 


5.96 The Department does not have the systems and practices in 
place to know whether it is delivering its services cost-effectively. It is 
not using financial and human resource information to its full extent 
for departmental analysis, to inform senior management, to improve its © 
strategic planning, or in discussions with client departments. Without 
key corporate work volume data, and financial and human resource 
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management information that is collected, analyzed, and reported to 
senior management on a regular basis, the Department does not know 
if it is delivering legal services cost-effectively. 


9.97 We found that the Department has the key systems and practices 
in place to effectively manage the delivery of legal services to meet the 
needs of government; however it has not worked with departments to 
help manage the demand for legal services and:thereby reduce costs. 
The current financial arrangements with client departments have few 
incentives to manage the increasing demand for legal services and 
costs. In addition the administration of the numerous cost recovery 
arrangements is inefficient. The Department has been aware of these 
problems for a number of years and, despite much effort on its part, we 
found no progress on this issue. 


9.98 In our opinion, the lack of a senior executive position with 
delegated authority for management practices has limited the 
Department’s ability to focus on and implement the improvements 
necessary to manage a large department responsible for delivering 
cost-effective professional services. 
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About the Audit 


Objectives 


Our objective was to determine whether 


- the Department of Justice Canada (Justice Canada) has the systems, practices, and processes to 
provide assurance that it is delivering quality at a level commensurate with the assessed risk and that it 
is managing the delivery of legal services cost-effectively; and 


- the Department has key systems, practices, and processes in place to effectively manage the delivery of 
legal services to meet the needs of government. 


Scope and approach 


The scope of our audit was the delivery of legal services to government provided by Justice Canada. 
These services include providing legal advice on departments’ policies and programs, drafting new and 
amending existing legislation and regulations, and representing the government in court—whether in 
defence of civil litigation or to enforce Canada’s laws through federal prosecutions. 


Provisions of the Federal Accountability Act came into force on 12 December 2006. This act created the 
independent Public Prosecution Service of Canada to prosecute offences under federal law. The new 
Public Prosecution Service of Canada is separate from Justice Canada and reports to the Attorney 
General. At the time of our detailed audit work, these prosecution services were provided by the Federal 
Prosecution Service—a branch of Justice Canada—and were included in our audit scope. Our findings 
and recommendations now relate to the work of the Public Prosecution Service of Canada as well as 
Justice Canada. 


We conducted our audit of the delivery of legal services to meet the needs of government through three 
general approaches—interviews, document reviews, and case file reviews. We also interviewed officials in 
other legal jurisdictions and in large private sector organizations with sizeable legal services units. 


To assess the management of resources for legal services we interviewed senior officials and collected and | 
assessed strategic and business planning documentation from across the Department. We examined the 
Department’s reporting to Parliament over the past three years. To examine the planning and management 
of resources on a case-by-case basis, we selected a purposeful sample of litigation files on the basis of 
specific criteria. We stratified files based on the six regions and the Department’s defined risk ratings. 

We selected files from three regions (British Columbia, Ontario, and the Prairies) to allow examination of 
a range of management styles and risk levels assessed by the Department (high and low). Files were 
selected evenly across a region with slightly more files in the higher risk category. Files were randomly 
sampled within the strata. We reviewed 43 of the 45 randomly selected files, as the Department did not 
have 2 of the files. 


‘To assess the quality management processes, we interviewed management from Justice Canada and 

8 client departments (Canada Revenue Agency, Canadian Food Inspection Agency, Citizenship and 
Immigration Canada, Fisheries and Oceans Canada, Health Canada, Human Resources and Social 
Development Canada, Indian and Northern Affairs Canada and Transport Canada). The 8 were selected 
from 42 client departments, based on the complexity of law required for their operations and their heavy 
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use and reliance on Justice Canada legal services. We reviewed documentation where it was available. 
We also examined and assessed quality management processes on a case-by-case basis. We selected a 
purposeful sample of files to review on the basis of specific criteria. The files were stratified based on eight 
legal service units; risk (high and low); and type of case (for example, appeal, litigation, or advisory). We 
selected low-risk or high-risk files in order to capture the variation of the practices in place to manage 
quality. We selected a purposeful sample of 91 files that were randomly sampled within the strata. 


To assess the Department’s knowledge of its costs and its rationale for the current cost recovery system we 
interviewed key officials on past, current, and potential future practices for cost recovery. We also collected 
relevant documentation and requested the agreements supporting the financial arrangements and cost 
recovery. We examined whether costs played a factor in decisions to proceed with litigation and appeals on 
a case-by-case basis. We selected a purposeful sample of files to review on the basis of specific criteria. The 
files were stratified based on the same eight legal service units, risk (high and low), and type of case (appeal 
and litigation). We selected a purposeful sample of 55 files that were randomly sampled within the strata. 
Two of these files were not included in our results, as the characteristics of the case were not applicable to 
our audit test, leaving 53 for which results are reported. We also selected five important initiatives 
designed to reduce the cost of litigation implemented over the past 10 years to assess the management of 
the initiative as well as the use of financial information and analysis. 


‘To assess the selection, appointment, and management practices for legal agents, we interviewed 
management and reviewed documents. In addition, we conducted a file review of the legal agents to 
examine their procurement process and supervision. These results should not be applied or generalized to 
the entire population of 7,282 files of the agents and firms selected between fiscal years 2002-03 

and 2006-07. Since there are two different types of legal agents, two different samples were selected: 
Federal Prosecution Services (FPS) and civil litigation. For FPS, we selected three of the six regions— 
Ontario (Toronto), Quebec, and British Columbia—to capture differing management practices. In 
addition, two levels of dollar values were chosen (less than $250,000 and $250,000 and greater) to review 
the level of supervision for higher value agent files. A total of 25 firms were randomly selected within the 
strata. For civil litigation agents, we randomly sampled 21 cases, selecting the same three regions. 
However, three dollar-value levels were selected: under $25,000, $25,000 to $250,000, and over $250,000, 
because the contracts in place were more evenly distributed across these ranges. We sampled a greater 


number of agents in the middle dollar range from $25,000 to $250,000. 


Criteria 
We expected to find the following: 
* The Department has assurance that it has an appropriate and effective quality management 
framework in place. 
* The Department has the systems and practices in place to effectively manage resources and align them 
with the needs of government for legal services. 
- The Department effectively manages the use of legal agents. 
* The Department has appropriate cost management information that it uses to manage the delivery of 


legal services efficiently and economically. 
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» The Department has assurance that its cost recovery system is the appropriate mechanism for 
providing legal services to government in a cost-effective manner. 


- The Department develops and implements alternatives to litigation and enforcement as a means to 
reduce the potential future cost of legal services. 


Audit work completed 


Audit work for this chapter was substantially completed on 5 December 2006. 


Audit team 


Assistant Auditor General: Hugh McRoberts 
Principal: Gordon Stock 
Director: Mary Riopelle 


Maryanna Basic 
Isabelle Dupuis 
Lori-Lee Flanagan 
Steven Mariani 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 5. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Progress since our 1993 audit 


5.13 Justice Canada should complete 
its corporate strategic planning and 


develop strategies to meet its long-term 


goals. (5.9-5.12) 


5.20 Justice Canada should examine 
the weaknesses noted in its 
management of legal agents used for 
civil work and take corrective action. 


(5.14-5.19) 


5.25 In addition to its current 
management of alternative dispute 
resolution initiatives, Justice Canada 
should ensure that it clearly defines 
each initiative’s objectives, estimates 


the potential cost savings and formally 


reviews the outcome of initiatives 
against their objectives. (5.21—5.24) 
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The Department agrees with the Auditor General’s 
recommendation. The Department created the Office of 
Strategic Planning and Performance Management in 

February 2005 to begin to strengthen the Department’s strategic 
planning and performance management capabilities, including 
integrating planning and strategic analysis, and performance 
measurement and reporting for the Department. The 
Department is also developing a corporate human resources 
strategy, which encompasses the key strategic human resources 
priorities of the Department as well as other human resources 
management initiatives in support of government-wide 
priorities. The Department will continue its efforts to implement 
planning frameworks and processes that reflect the unique needs 
and challenges that it faces in meeting its strategic outcomes. 


The Department agrees with the Auditor General’s 
recommendation. In fact, the Department has an action plan to 
support the continued implementation of improvements to the 
management of legal agents. This action plan will also address 
the issues raised by the Auditor General. 


The Department agrees with the Auditor General’s 
recommendation, which is consistent with the recent review of 
the Department’s dispute resolution initiative that has resulted 
in a new mandate focused on providing services to departments 
supporting long-term prevention and early resolution of disputes. 
These client-focused services include systems design, legal 
advisory support, and training, all of which are linked to the 
departmental priorities of improving practice management and 
managing the volume of litigation. 
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5.29 Justice Canada should define == The Department agrees with the Auditor General’s 

performance measures for its corporate | recommendation. As part of its departmental performance 

priorities and improve its performance report to Parliament, the Department will incorporate relevant 

reporting to Parliament. (5.26—5.28) measures from the annual assessment of our Management 
Accountability Framework completed by the Treasury Board 
Secretariat. 


More broadly, the Department has implemented a number of 
changes to its annual reports on plans and priorities and 
departmental performance reports in an effort to improve 
reporting to Parliament and the public on the results of 
departmental initiatives and priorities. Also, as part of its efforts 
to improve reporting, the Department has implemented a 
practice whereby a committee of performance measurement 
specialists within the Department, as well as one external 
adviser, assess the quality of data used to arrive at management’s 
assessments of the degree to which the Department is achieving 
its priorities each year. The results of these reviews are reported 
in the departmental performance report. 


Quality assurance 


9.62 Justice Canada should developa § The Department agrees with the Auditor General’s 


quality management system or systems recommendation and will build on the elements of its quality 
for each of its practices to provide work recognized by the Office of the Auditor General with a 
senior management with ongoing and view to having a sound quality management approach. 


reliable assurance that all services meet 
established minimum quality standards. 


(5.30-5.61) 
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Resource management 

5.93 Justice Canada needs to make The Department agrees with the Auditor General’s 
improvements to its resource recommendation. The Department of Justice Canada will 
management. The Department should continue the work underway to develop approaches to further 


develop its capacity to use management information to enhance 
management decisions, and regularize its cost recovery 
agreements with government departments. 


* use management information on 
workload volumes and costs for all of 
its legal services to better monitor 


trends, plan, set priorities, manage The Department is currently working on putting in place an 
resources, analyze service delivery initiative to better coordinate and integrate its law management 
options, and identify opportunities for practices. The Department will, at the same time, review its 
controlling costs; management practices to identify and put in place corporate 


reports to provide management information to support 
departmental decision making. Senior management authority to 
lead our work on management practices will be strengthened. 


* work with client departments to 
effectively plan, manage, and improve 
the cost-effectiveness of their 
financial arrangements with a view to 
managing demand for legal services 
and thereby reducing costs; and 


* ensure that a senior manager with the 
necessary authority leads the 
improvement of management 
practices at the Department and 


oversees implementation of these 
changes. (5.63—5.92) 
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Chapter 


Modernizing the NORAD System in 
Canada 


National Defence 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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Main Points 


Introduction 


Background 
Focus of the audit 


Observations and Recommendations 


Approval to modernize the NORAD system 


There were early indications that the modernization project was in difficulty 

National Defence had spent $65 million when the contract was terminated 

National Defence chose to unilaterally develop its own air surveillance and control system 
The Department stopped working on the MASE system to join a new US-led option 
Replacing the NORAD system plus related expenses brings total costs close to $156 million 
Anticipated savings have not been realized 


Project management and oversight 


The Statement of Operational Requirement has not been updated since 1997 
Some key project management components have not been completed 
National Defence did not inform the government of project scope changes in a timely fashion 


Oversight would have increased by designating the modernization project a Major Crown Project 


The Department did not take advantage of already existing controls to provide better oversight of this project 


Construction of the above-ground complex 


Costs and design have grown considerably since first approved 
Use of the building is limited by design and construction irregularities 


Conclusion 
About the Audit 


Appendix 


List of recommendations 
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Main Points 


Canada and the United States are partners in the air defence of North 
America under the North American Aerospace Defense (NORAD) 
agreement. Under this agreement, Canada and the US operate an 
integrated and interoperable air surveillance and control system to 
secure North American airspace. In 1997, the two countries agreed to 
upgrade and modernize this system. Originally, the Canadian 
government approved funding for about $93 million to modernize the 
Canadian portion of the NORAD system—the Canadian Air Defence 
Sector Air Operations Centre. Since the original approval for 
modernization, the project has undergone several changes and cost 
increases. We examined how the modernization has progressed, what it 
has cost National Defence, and what has been delivered. 


Heightened security demands since September 11, 2001, have put 
emphasis on the need to modernize the NORAD surveillance and 
control systems. National Defence has a significant investment in the 
modernization project and needs to implement it in a way that is 
cost-effective and ensures that operational requirements are met. In 
May 2006, Canada and the United States renewed the NORAD 


agreement to continue operations in both countries. 


¢ Modernization was originally expected to cost National Defence 
$87 million, plus about $6 million for a definition phase, and was to 
be completed by 2001. However, we estimate that the Department 
has spent about $125 million so far, and bringing the modernization 
to completion is expected to cost another $18 million. Other related 
expenses of about $13 million will bring the expected total cost to 
about $156 million. The project took longer than planned to deliver 
and has cost more than initially expected because of problems with 
system development and project management. Recently, the 
Department successfully installed a new air surveillance and control 
system at its sector air operations centre, and it reports that the 
system is meeting initial operating requirements. However, 
anticipated savings of up to $16 million a year in personnel and 


operating costs have not materialized. 
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Neither National Defence nor the government made appropriate use 
of mechanisms available for managing large, high-risk projects like 
this one. Early signs that the project was in trouble and that costs 
were climbing did not prompt moves to strengthen its oversight. We 
asked why, for example, a decision was not made at the time to 
designate this a Major Crown Project. Treasury Board Secretariat 
officials initially declined to provide us access to the working papers 
on this issue because they viewed these as cabinet confidences. At 
the conclusion of the audit and after a search, Treasury Board 
Secretariat officials assured us that there were in fact no Treasury 
Board Secretariat working papers to provide to us. 


Information provided to the government for increased project 
funding was not always timely or accurate, and did not always reflect 
the risks of the project or how its scope had changed. 


National Defence entered into an agreement with the United States 
Department of Defense as an equal partner, during the development 
phase of the first effort to implement a new air surveillance and 
control system, and on that basis it agreed to pay 50 percent of 
common costs. Yet it could not provide us with evidence to 
demonstrate that it had in fact shared control of the project. Later, 
problems appeared in the Department’s ability to control its own 
development of a second, replacement air surveillance and control 
system, which it was working on for the Canadian Air Defence 
Sector. Work on this second system was stopped in favour of a third 
system developed for NORAD, mainly with US Department of 
Defense input into the requirements. This air surveillance and 
control system has now been installed and is operating at the 
Canadian Air Defence Sector air operations centre. Future 
modernization and upgrade work is planned, but National Defence 
still has not clearly set out its own Statement of Operational 
Requirement and needs to review requirements before continuing. 


National Defence has responded. The Department agrees with our 
recommendations and is proposing actions to address concerns. 
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Introduction 


6.1 Canada shares the defence of North American airspace with the 
United States through the North American Aerospace Defense 
Agreement (NORAD). NORAD is the foundation for Canadian—US 
aerospace defence cooperation. Its primary mission includes 
surveillance, detection, monitoring, validation, and warning of air 
attacks against North America. Since it was formally established 

in 1958, NORAD has evolved to meet a growing variety of security 
threats. In 1991, the NORAD mission was expanded to include 
counter-narcotics air surveillance operations. After the September 11, 
2001 terrorist attacks, NORAD has increased its role in the security of 
internal domestic airspace. 


6.2. NORAD is a combined command established by mutual 
agreement between Canada and the United States. Based on available 
information, NORAD provides warning and assessment of air threats 
to each nation’s responsible authorities. 


Background 


6.3. NORAD air defences are built around a network of radars 
connected to a command and control system for tracking targets and 
deploying fighter jets to intercept and, if necessary, engage threats. 
NORAD command and control is based at the Cheyenne Mountain 
Operations Center in Colorado Springs, Colorado. Surveillance and 
control of North American airspace is conducted from Canada, the 
continental United States, and Alaska. Exhibit 6.1 shows the NORAD 
regions and sectors. The Canadian NORAD Region (CANR) 
Headquarters is in Winnipeg, and the sector air operations centre is at 
Canadian Air Defence Sector (CADS) in North Bay, Ontario, which 


provides the tactical command and control for Canadian airspace. 


64  Acrucial element of NORAD’s network of sector air operations 
centres is the computerized air surveillance and control system. It 
displays data gathered from sensors located across North America and, 
when they are deployed, from surveillance and reconnaissance aircraft. 
Each sector’s air operations centre relies on the computerized system 
to assist in processing information, identifying targets, and guiding 
interception to the targets. Critical information generated in North 
Bay is forwarded to CANR Headquarters in Winnipeg, and to the 
Cheyenne Mountain Operations Center in Colorado Springs. 


6.5 NORAD began operating the AN/FYQ-93 (Q-93) system 


in 1983, at which time it was expected to last 20 years. In the 
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Exhibit 6.1 NORAD regions and sectors 


Alaskan 
NORAD Region 


(ANR) 
Alaskan Air Defense Sector 
e 


Canadian NORAD Region 


(CANR) 
CANR Headquarters 
Winnipeg 
6 
bd Canadian Air Defence Sector 
Western Air Defense Sector (CADS) 
(WADS) ° 
~ ® 
© Eastern Air Defense Sector 
: (EADS) 
Cheyenne Mountain Continental US 
pec NORAD Region 
(CONR) 


NORAD is divided into three regions—the Canadian NORAD region, the Continental 
NORAD region (US) and the Alaskan NORAD region. 


early 1990s, National Defence and the US Air Force began planning 
how to replace the Q-93 system at the end of its 20-year life span. As a 
partner in NORAD, Canada was involved in the decision to modernize 
and replace the system. However, initial plans and project scope for 
modernizing the NORAD computer system were not realized, and 
since that time, National Defence has been involved in developing two 
other replacement systems, one of which—the Battle Control System— 
Fixed or BCS-F 


has recently become operational. 


6.6 As part of the modernization, National Defence expected to 
generate savings. Some savings would be realized through a reduction 
in the number of personnel required to run the new air surveillance 
and control system. Further savings would be realized through a move 
from the existing underground complex in North Bay to a new, 
above-ground complex that would house the modernized system. 


Since 1995, costs to National Defence for modernizing the NORAD 
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computer system have amounted to about $125 million in spending on 
system development and new facilities. The Department estimates that 
another $18 million in expenses will be necessary to bring the project 
to completion, plus about $13 million in related costs. 


Focus of the audit 


6.7 = Our audit examined the National Defence components of the 
project to modernize NORAD’s Q-93, the air surveillance and control 
system, to determine whether capability improvements had been 
developed and installed, and whether they met Canadian capability 
requirements. The audit examined project management, costs, 
timelines, and what was delivered. We also examined whether project 
activities complied with government policies, directives, and 
guidelines. 


More details on the audit objective, scope, approach, and criteria are 
in About the Audit at the end of this chapter. 


Observations and Recommendations 


6.8 The North American Aerospace Defense (NORAD) Canadian 
Air Defence Sector Air Operations Centre has recently installed the 
newly developed Battle Control System—Fixed (BCS-F), an air 
surveillance and control system. The contractor for BCS-F began 
installation at North Bay in early 2006, and the Canadian Air Defence 
Sector Air Operations Centre switched from the old system (Q-93) to 
BCS-F operations in October 2006. 


6.9 BCS-F represents the third attempt by National Defence to 
install a modernized air surveillance and control system in the sector 
air operations centre, a process that has spanned almost 12 years. 
Work to replace the old system began in 1995 when the government 
approved $6.4 million in funding for National Defence to participate, 
in cooperation with the United States, in the definition phase of the 
Region/Sector Air Operations Centre Modernization Project. In 1996, 
a Memorandum of Understanding was signed between National 
Defence and the United States Department of Defense to jointly 
modernize NORAD’s air surveillance and control system. The aim was 
to develop a common NORAD system that could improve the 
coverage of North American airspace. Throughout the project, the 
two countries would fund an equal share of their common costs; they 


would also have a shared software maintenance strategy. As equal 
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partners, Canada and the US were to share decision making, data 
rights, and industrial participation. 


6.10 National Defence personnel and US Air Force personnel were 
assigned to the project management office to ensure that each 
partner’s concerns and specific requirements were represented. A 
bi-national steering committee was set up to review progress and 
resolve any issues that might arise throughout the life of the project. 


6.11 In February 1997, after completion of the project definition 
phase, the government approved the Department's request to spend 
$59.5 million on developing a new system. It was expected that the 
new system would be in place by September 1999 and that work to get 
it to full operational capability would continue until fall 2001. The 
Department supported its request for funding approval by showing that 
it expected to save about $16 million per year in operating costs. 
Savings were to begin around 2004, at which time the Department had 
planned to shut down its underground complex. The Department also 
anticipated a change in operations, with a reduction in personnel. 


6.12 A few months later, the government gave approval, with 
conditions, to spend an additional $27.5 million on the following: 


* a deployable capability study; 
* an upgrade of the internal communications system; and 


* construction of an above-ground complex to house the new air 
surveillance and control system, allowing National Defence to 
close the underground complex. 


6.13 In total, it was expected that the modernization would cost 
National Defence about $87 million, in addition to the $6.4 million 
spent on project definition, but that the possible savings would allow 
the Department to have its investment paid back within four to 

five years after project completion. 


6.14 In March 1997, a contract to develop the proposed new air 
surveillance and control system for NORAD was awarded to a US 
company, and development work began. Although a system concept 
was defined at this point in time, system development components and 
costs had not yet been clearly set out. 


There were early indications that the modernization project was in difficulty 


6.15 In early 1997, National Defence told the government that the 
risk of not completing the project was low. However, by July 1998, it 
became apparent that costs were escalating. The bi-national steering 
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committee was informed that projected costs for completing the 
modernization project were increasing. National Defence personnel, 
who were monitoring costs and overseeing progress, now considered it 
a high-risk project. 


6.16 In February 1999, the Department requested an increase in 
funding to complete the modernization project. The Department 
reported that it was very confident that additional funds would enable 
the project to achieve operating capability, even though there were 
concerns at that time about the contractor’s slow progress and 
escalating costs. National Defence received approval for an additional 
$36.7 million for this project, bringing the total expected cost for the 
modernization to about $130 million at that time. 


6.17 We noted that the government had expected the Department to 
demonstrate progress on the modernization project and that funding 
would be approved in two phases. Funding for the second phase was 
conditional upon successfully achieving the first phase. However, 
because of delays and cost overruns, the government approved a 
change in project scope so that money set aside for phase two could be 
spent on continuing phase one in order to achieve core operational 
requirements, even though there was no evidence that the first phase 
was successfully achieving objectives. Hardware and software modules 
were not ready and significant development remained to be done. 


6.18 In October 1999, despite assurances from the Department that 
increased funding would enable the modernization project to succeed, 
the contract for developing a new system was cancelled by the US 
Department of Defense. Concerns that progress was slow, along with 
cost projections that were more than triple the initial expected project 
completion costs, led the US Department of Defense to halt any 
further work. 


National Defence had spent $65 million when the contract was terminated 


6.19 National Defence has estimated that it had spent $65 million on 
this contract and on other related expenses by the time the project was 
terminated. Our review found that the Department had paid about 
$57 million for contractor development work, and that the 
Department’s other project management costs brought the total to 
about $65 million. Despite its $65-million investment, the Department 
was left with the same system and operations that it had had before. 
The Department did receive about $1.5 million in hardware, as 
stipulated in the contract. Some of this hardware was used for training, 


and some remained in storage. 


- — 
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6.20 Department documents indicated that despite its 50-50 
partnership in this project, and notwithstanding concerns about its 
difficulties, National Defence did not believe it could stop the process 
sooner or pursue an alternative course of action. 


6.21 During this time, National Defence was also upgrading its 
internal communications system. The Department put that work on 
hold when the air surveillance and control system development 
contract was terminated. It had spent about $7.7 million on upgrades 
when the work was stopped. The Department was able to derive some 
benefit from this previous work when the internal communications 
systems work was later continued. 


National Defence chose to unilaterally develop its own air surveillance and control 
system 


6.22 After the contract was terminated in 1999, NORAD officials 
determined that the AN/FYQ-93 (Q-93) air surveillance and control 
system would not be replaced immediately, and that NORAD could 
continue to use it until 2009. Nevertheless, National Defence 
informed NORAD that it needed to save money, and that it therefore 
still planned to move its operations to a new above-ground complex, 
reducing personnel in the process. Although the Memorandum of 
Understanding with the US stated that both countries would jointly 
develop an interoperable air surveillance and control system, National 
Defence decided to develop and install its own system in the 
above-ground complex, as a short-term solution until a NORAD-wide 
system could replace it. In part, National Defence questioned the 
long-term viability of the Q-93 system at its sector air operations 
centre, and was concerned that a lack of spare parts might prevent 
Q-93 from remaining operational and reliable until 2009. 


6.23. At the time of our audit, we found no issues related to spare parts 
shortages affecting Q-93 system operations. An analysis conducted by 
the Department in 2000 forecast that enough spare parts would be 
available to keep the system operational until 2010. 


6.24 In September 2000, National Defence chose NATO's Multiple 
AEGIS Site Emulator (MASE) system. It planned to modify MASE to 
be compatible with the Q-93 system, which was still in use in the US. 
The Department began working on MASE system development and 
entered into agreements with other agencies to modify the system. 
However, it did not submit a request for increased spending authority 
to the government until June 2003, nor did it seek approval to change 
the scope of the project. We found that internal delays slowed the 


8 ] Chapter 6 


Report of the Auditor General of Canada—May 2007 


Report of the Auditor General of Canada—May 2007 


MODERNIZING THE NORAD SysTEM IN CANADA—NATIONAL DEFENCE 


approval process. The Department initially expected to have a viable 
system ready by fall 2003, but by the time the submission was made to 
the government, the date had slipped to July 2004. 


6.25 The Department supported its MASE-led approach with possible 
reductions of about 30 to 40 percent in operations and maintenance 
staff. The Department forecast that these reductions would lead to 
savings on the order of $10 to $12 million annually. It told the 
government that modifying the MASE system was a medium- to 
high-risk strategy for replacing the Q-93 system. In June 2003, the 
government gave approval to increase the funding to about 

$143 million (GST excluded). This was done in order to allow 
modification work to continue on the MASE system and to provide 
funding for other ongoing project requirements, such as the 
construction of the above-ground building. The government stipulated 
at this time that National Defence was to begin submitting annual 
reports on 


* progress, 
* changes in costs and schedules, and 


* updates on how risks were being managed. 


6.26 Exhibit 6.2 shows the project costs that were expected as of 
June 2003. 


Exhibit 6.2 Expected Project Costs, June 2003 


| Initial Estimated Cost 1997 June 2003 Estimated Cost 
Project definition $6.4 million $6.4 million 
System replacement $59.5 million 
$136.6 million 
Canadian unique requirements $27.5 million 


$143.0 million 


$93.4 million 


The Department stopped working on the MASE system to join a new US-led option 


6.27. Although it was working to modify the Multiple AEGIS Site 
Emulator (MASE) system, National Defence remained committed to a 
bi-national air surveillance and control system for NORAD. Shortly 
after receiving approval for its MASE-led solution, National Defence 
was approached by the US Air Force to participate in the development 
of the Battle Control System— Fixed (BCS-F) as the NORAD-wide 
system. In January 2004, senior management decided that the 
Department would join the BCS-F option, but would continue 
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working on the MASE system as a fallback, should BCS-F prove 
unworkable. However, by mid-2005, it had become clear that problems 
with modifications to the MASE system would be difficult to overcome 
and that target dates would not be met; however, BCS-F was 
progressing. Therefore, National Defence stopped work on the MASE 
system to instead pursue BCS-F as the system for NORAD. It was 
expected that the BCS-F system would be installed, tested, certified, 
and operational at the sector air operations centre by early 2006. 


6.28 The Department estimated that it would spend about 

$13 million to install and test the BCS-F system. It did not request an 
increase in spending authority from the government, because MASE 
system expenses were proving to be considerably lower than initially 
anticipated, and because the Department believed that, with the funds 
remaining, BCS-F system costs could be absorbed into the already 
approved $143 million. 


6.29 The Canadian Sector Air Operations Centre began using the 
BCS-F system in October 2006, and is currently operating that system 
with the rest of NORAD. 


Replacing the NORAD system plus related expenses brings total costs close to 
$156 million 


6.30 At the time of our audit, we found that the Department had 
spent about $125 million of the approved $143 million for this project. 
Further expenses of about $18 million were expected for ongoing work 
to finalize this part of the modernization project. We found other costs 
that had been incurred by the Department as a direct result of this 
project, but that had been funded separately. These costs were 
incurred at several stages: during construction of the above-ground 
building at North Bay, at the time that the systems were moved from 
the underground facility to the above-ground building, and after the 
system was installed. Ensuring security of the facilities was expected to 
incur further expenses. However, no cost estimate was available to us 
at the time of this audit. We estimate that total costs for modernizing 
the NORAD system in Canada will amount to about $156 million. 


Anticipated savings have not been realized 


6.31 Despite expectations that there would be savings of as much as 
$16 million annually by 2004, at the time of this audit, the 
modernization project had not been able to demonstrate that any 
savings have been made or will be made. The Department based its 
projected savings on analyses that included closing the underground 


10 


Chapter 6 


Report of the Auditor General of Canada—May 2007 


Project management and oversight 


Statement of Operational Requirement— 

A mandatory National Defence project document 
that describes the characteristics of the 
operational requirements to be used by technical 
and procurement staff. It contains critical 
performance criteria for evaluating technical 
options and system performance 


MODERNIZING THE NORAD System IN CANADA—NATIONAL DEFENCE 


complex, building a small above-ground operations centre, and 
reducing personnel requirements. However, at the time of our audit in 
early 2007, we found that 


* the underground complex remained open for operational reasons, 
thus incurring the expenses necessary to keep it functional and 
safe; 


* the above-ground complex has grown from the small operations 
centre envisioned in 1998 to the current larger two-storey 
building; and 


* the Department has not yet been able to determine whether it will 
have any reductions in operational personnel under the new 
system. 


6.32 Recommendation. National Defence should update its analysis 
of the NORAD system modernization project’s expected savings, and 
should determine when such savings will be realized. 


The Department’s response. Agree. In response to a meeting of the 
Senior Review Board conducted on 11 December 2006, the business 
case for the Region/Sector Air Operations Centre Modernization 
Project was updated on 15 February 2007. As a significant portion of 
the original business case was predicated on the possible relocation of 
the operational elements, much of the infrastructure-related savings 
originally envisioned will not be realized. The current savings 
projections are approximately 18 percent of the original 1996 business 
case estimate; further studies on the actual savings are ongoing. 


Although the underground complex has been declared surplus and is 
currently moving through the surplus property process, two additional 
challenges have impacted plans for closure of the underground 
complex. First, the underground complex has been identified as a 
heritage site. This will entail providing a minimal level of support to 
the underground complex until a permanent solution is found. Second, 
due to residual security concerns, a small number of systems have yet 
to be moved from the underground complex. This should be rectified 


over the next few months. 


The Statement of Operational Requirement has not been updated since 1997 


6.33 In 1997, National Defence drafted a Statement of Operational 
Requirement that outlined its needs for modernizing the North 
American Aerospace Defense (NORAD) system. A Statement of 
Operational Requirement is a mandatory document for all National 


Defence equipment projects. It defines expectations and guides 
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decision making. National Defence policy requires that statements of 
operational requirement be updated as a project progresses and as 
requirements change. We expected that the Department would have 
reviewed the 1997 Statement of Operational Requirement as its policy 
requires, and before proceeding with other options. 


6.34 However, we found that the Statement of Operational 
Requirement document has not been updated since 1997, even though 


¢ NORAD’s responsibilities have changed since 1997; 


* advances in information technology since 1997 may have changed 
the risks or created new opportunities that are not addressed in 
the original Statement of Operational Requirement; and 


* the bi-national partner for this modernization, the US Air Force, 
has twice updated its version of a Statement of Operational 
Requirement. 


6.35 We found that because National Defence had proceeded with 
the Multiple AEGIS Site Emulator (MASE) system development and 
with acquisition of the Battle Control System—Fixed (BCS-F) system 
without first re-examining its own Statement of Operational 
Requirement, it did not have an updated assessment of its needs. It was 
therefore unable to provide us with assurance that the systems would 
perform as needed, and was also not positioned to evaluate future 
system performance. As modernization and upgrades continue for the 
Canadian Air Defence Sector, an updated Statement of Operational 
Requirement is a key element for guiding the department’s further 
analysis and decision making. 


6.36 Recommendation. National Defence should ensure that any 
further modernization and upgrade of Canada’s air surveillance and 
control system is supported by a current and clearly defined Statement 
of Operational Requirement. 


The Department’s response. Agree. Canada and the US have 
continued to evolve the system modernization requirements over time. 
With the full participation of Canada in earlier NORAD requirements 
documents, our requirements were incorporated into the Battle 
Control System Operational Requirements Document and the System 
Capabilities Document. In addition, as the Department determines the 
need for follow-on effort, all capability deficiencies will be validated as 
part of the Statement of Requirements for the Canadian Air Defence 
Sector Upgrade Project. 
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Some key project management components have not been completed 


6.37 We examined the process followed by the Department to manage 
modification of the MASE system. The Department had entered into 
agreements with other agencies to modify the software, but did not 
identify start and end dates for the work. As a result, the work was not 
treated as a priority by the other agencies, despite National Defence 
timelines, and project completion was postponed. 


6.38 We examined whether the Department had analysed options and 
assessed cost and feasibility. Government policy, along with generally 
accepted good management practices, require that both the expected 
benefits of any system and its estimated life cycle costs be analysed and 
included in the decision-making process. While few options may have 
existed for National Defence as a NORAD partner other than to 
accept the BCS-F system as its sector air operations centre system, we 
nevertheless expected the Department to have identified system 
benefits, as well as any associated risks, and to have assessed future 
costs. The Department could not provide us with evidence that it had 
done this analysis. 


6.39 We expected that the Department would have a business 
continuity plan for its system operations, as required under the 
Government Security Policy. A business continuity plan is developed 
by assessing the threats and risks that could disrupt critical services. It 
identifies what needs to be done to protect against system failures. 
Although we did observe that some steps had been taken by the 
Department to protect against events such as a major power failure, we 
also expected to see a planned response, on the part of the 
Department, to a major disruption. Such a planned response would 
identify which systems take priority and the need for system back-up 
and system redundancy. National Defence was unable to provide us 
with a business continuity plan that addressed these key elements. 


6.40 Recommendation. The Department should conduct a threat 
and risk assessment for the air surveillance and control system, to 
determine the extent to which the Department will provide system 
back-up and redundancy, and to develop a business continuity plan as 


necessary. 


The Department’s response. Agree. The Department agrees that 
threat risk assessments and business continuity plans are important 
and necessary. For the Region/Sector Air Operations Centre project, a 
threat risk assessment was prepared for the first system and revised for 
the MASE system. For the current BCS-F system, a threat risk 
assessment is being performed as part of the Department of Defence 
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Information Technology Security Certification and Accreditation 
Process. Within this certification and accreditation process, back-ups, 
redundancy, and continuity will all be addressed. The requirement for 
site redundancy will be further examined under the Canadian Air 
Defence Sector Upgrade Project. 


National Defence did not inform the government of project scope changes in a 
timely fashion 


6.41 Government policy requires departments to seek authority from 
government before changes are made to a project in areas that affect 
the project’s scope, such as key deliverables, risk management, timing, 
contracting strategy, special requirements, and project management. 
We found that over the course of the modernization project, the 
Department did not officially inform the government in a timely way of 
significant changes in some of these areas. 


6.42 We found that although government approval of the project 
called for a joint effort to share in the development, acquisition, and 
software maintenance of a new air surveillance and control computer 
system common to Canada and the US, the Department acted 
unilaterally to modify the MASE system for its own use. The decision 
by National Defence to change its key deliverable and adapt the 
MASE system was made in 2000. However, the Department did not 
officially inform the government of this decision until 2003. The 
Department has informed us that government officials were 
“informally” notified in October 2000 of this change. 


6.43. We examined whether this decision resulted in a change to the 
deliverable, the timing, and the contract strategy. Government officials 
informed us that the decision was considered to change only the 
delivery method. They stated that since the ultimate goal was to 
replace the Q-93 system, the decision to adopt MASE did not 
represent a change in project scope that would require government 
approval. It is our view that National Defence did, in fact, change the 
scope of the project, from developing a bi-national system in common 
with the US, to implementing a short-term solution for Canada alone. 


6.44 Despite the fact that, according to the Department, the project’s 
risks were changing, we were not provided with evidence that this was 
communicated to the government in a timely way. 


6.45 The Department had estimated that implementing its MASE 
system solution would cost an additional $9.5 million. Officials 
maintained that the Department retained the authority to proceed 
with the implementation, because costs were within the already 
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approved spending authority (that is, within the amount remaining 
after approximately $65 million had been spent on the cancelled 
contract), and because the Department’s approach still fell within the 
general scope of modernizing the NORAD system. Ultimately, 
expenses were considerably lower than planned—about $440,000— 
due to problems delivering the MASE modifications. 


6.46 In the Memorandum of Understanding signed with the US, 
Canada was to be an equal partner in the joint development of an 
interoperable air surveillance and control system, sharing the common 
costs, the decision making, the data rights, and the participation of 
industry. When the Department joined in the acquisition of the BCS-F 
system, it did not inform the government of this change in scope from 
being a partner to becoming a customer buying a product developed 
mainly from US Air Force stated operational requirements. 


Oversight would have increased by designating the modernization project a Major 
Crown Project 


6.47 Given the cost escalations, delays, and risks, we found that 
sufficient warnings were available to the Department and the 
government to consider designating the modernization project a Major 
Crown Project. Government policy on Management of Major Crown 
Projects identifies costs, risks, and a department’s capacity to manage a 
project as key factors in determining whether it should be designated a 
Major Crown Project. Such a designation would have required 
National Defence to conform to a more defined and rigorous structure 
for reporting to the government. In 1998, National Defence officials 
asked Treasury Board Secretariat officials whether the modernization 
should be designated a Major Crown Project. The modernization was 
not designated a Major Crown Project, but Treasury Board Secretariat 
officials initially declined to provide us with their working papers 
setting out the reasons for this decision, because they viewed these as 
cabinet confidences of a type that were excluded from our right to 
access under a 1985 Order in Council. At the conclusion of our audit, 
Treasury Board Secretariat officials assured us, following a search, that 
working papers on this topic did not in fact exist. Accordingly, we are 


unable to assess the rigour and quality of the government's oversight. 


6.48 National Defence management policy states that increased 
senior management scrutiny is required for Major Crown Projects. 
Although this project was under scrutiny by a senior review board at 
the Department, we found that the board met only once a year, which 
is the minimum requirement under the Department’s defence 
management system. Had the modernization been designated a Major 
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C4ISR stands for command and control, 
communications, computing, intelligence, 
surveillance, and reconnaissance. National 
Defence created the C4ISR Oversight 
Committee in 1999 to respond to the need for 
more coordination of requirements. The 
Committee provides a strategic perspective and 
leadership on all C4ISR related matters. 


Construction of the 
above-ground complex 


Crown Project, the project would have been required to report to a 
Special Project Advisory Committee. As well, the Statement of 
Operational Requirement would have been passed to the Joint 
Capability Requirement Board (made up of the chiefs of the Army, 
Navy, and Air Force staff, along with the Assistant Deputy Ministers) 
for endorsement. 


The Department did not take advantage of already existing controls to provide better 
oversight of this project 


6.49 The Department’s Assistant Deputy Minister (Information 
Management) is the National Defence authority for information 
technology projects and initiatives. Each National Defence service 
(Army, Navy, Air Force) is responsible for development of projects 
categorized as weapons system projects, and each may ask the 
Assistant Deputy Minister (Information Management) for advice if 
need be. Because National Defence categorized the modernization 
project as a weapons system for sector air operations, the project’s 
development remained under the control of the Air Force. 


6.50 Although it is clear that the Air Force is best positioned to state 
what it needs for NORAD operations, the Assistant Deputy Minister 
(Information Management) is concerned with the development of all 
information technology systems in the Department and oversees how 
they integrate with each other. The Assistant Deputy Minister 
(Information Management) recommended that the C4ISR Oversight 
Committee monitor the modernization project, because the new air 
surveillance and control system would have to integrate with other 
defence systems, and recommended that the modernization project not 
be managed independently. Although senior management noted these 
concerns, the Department did not submit the project to the Oversight 
Committee as recommended. 


Costs and design have grown considerably since first approved 


6.51 National Defence originally planned to build an above-ground 
complex that was much smaller than the current building. Initially, in 
spring 1998, the Department proposed to the Minister that it would 
build a small (4,500 square metre) operations centre for about 

$6 million. Shortly thereafter, however, the Department recognized 
that it had underestimated space requirements in its initial business 
case. The Department then made revisions that expanded the size of 
the proposed building to about 7,000 square metres, a change that 
would raise the cost to about $9 million. Later, in October 1998, as 
part of an effort to reduce infrastructure at the Canadian Forces Base 
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New above-ground complex at Canadian Air 
Defence Sector, North Bay, Ontario 


Source: Department of National Defence 
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at North Bay, plans for the building changed. Originally conceived as a 
small operations centre, it would now be a larger (9,560 square metre) 
two-storey building large enough to house the majority of operations 
on the base. Costs for the new plans were estimated at $12 million. 
This estimate was later revised to $18 million. When the contract for 
the first air surveillance and control system development was cancelled 
in 1999, National Defence put the building construction on hold. 


6.52 In 2003, after the Department decided that it would install a 
modified version of the Multiple AEGIS Site Emulator (MASE) system 
as a temporary NORAD system, the Department started construction 
on the 9,560-square-metre complex. By this time, construction costs 
had increased to about $23.9 million, and we estimate that at the time 
of this audit the costs were close to $28 million. We also found other 
costs that resulted from the new construction, but that were not 
included in the Department’s building costs. Examples of such 
expenses are costs for moving systems into the new building and costs 
related to site security. We estimate that overall, National Defence will 
spend about $38 million on expenses related to the above-ground 
complex, for constructing it, moving into it, and addressing some of 
the facility’s other requirements. 


6.53 The Department supported moving into an above-ground 
building by stating that doing so would generate savings. The 
Department cited these potential savings as part of the rationale for 
pursuing development of the MASE system. However, we were not 
provided with evidence showing how savings were calculated or how 
costs would be reduced despite the growth in size of the building. 


Use of the building is limited by design and construction irregularities 


6.54 We found that planning and approval for the construction of the 
building had proceeded without a clear understanding of the 
operational requirements. Originally, the building was designed to 
accommodate the replacement for the old Q-93 system. Later, it was 
decided that the building would also house most of the base 
operations. As built at present, the above-ground complex cannot 
accommodate base operations as originally planned. There are now 
concerns about the extent to which the base can decommission other 


buildings and reduce its infrastructure. 


6.55 We found that the Department had not completed a review of 
the building security requirements prior to construction. National 
Defence requires that a security review checklist be completed for new 
buildings, to ensure that security concerns are identified and 
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addressed. Department officials told us that due to time and budget 
constraints, this step was not taken. Several security concerns did arise 
during construction that have led to questions about the building and 
the subsequent feasibility of operating in it. These questions 
specifically concern 


¢ the blueprints and purpose of the building, which were placed in 
the public domain when they were made available to interested 
contractors; 


¢ the physical control of the building and access to the site during 
construction, which were limited; and 


¢ the workers on site, none of whom were security cleared to work 
there. 


6.56 Asaresult, the Department is currently examining any extra 
steps it must take to ensure that the building can be used for the 
purposes intended, and to determine the necessary costs. At the time 
of this audit, the Department had not yet decided on its course of 
action, and hence was not able to provide us with this assessment. 


6.57 Asa result of concerns over the facilities in the above-ground 
building, the Department has had to maintain some operations in the 
underground complex longer than expected. At the time of this audit, 
despite plans to save costs by closing the underground complex and 
operating out of one facility above ground, National Defence was still 
operating both facilities. 


Conclusion 


6.58 Because of the way this modernization project was structured 
and managed, it has cost National Defence significantly more than 
originally planned. Decisions were made without complete information 
or an understanding of real savings. We found that projected savings 
are not being realized, and that costs for the construction of the sector 
air operations centre, and for developing and installing a new air 
surveillance and control system in the Canadian Air Defence Sector, 
are almost double the original estimate. 


6.59 We examined whether government policies, directives, and 
guidelines were followed by the Department as it replaced the old AN/ 
FYQ-93 (Q-93) system. We found that the Department did not report 
on project difficulties or changes in scope in a timely way. This, in our 
opinion, contributed to the modernization project having less 
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oversight than would be expected for a project entailing such cost, risk, 
and complexity. We found as well that some information presented to 
decision makers about the progress of the modernization project did 
not adequately portray the risks. There were early indicators that the 
project was experiencing significant cost overruns, and that it was 
failing to deliver the intended product—an operational air surveillance 
and control system. Sufficient indicators were evident to warrant more 
oversight by the Department and by the government. There were also 
several indicators that this project would have benefited from being 
designated a Major Crown Project. 


6.60 We also examined whether the new system met National 
Defence requirements for air surveillance, but we found that the 
Statement of Operational Requirement had not been updated to 
reflect a review of the current environment or of the options. While we 
recognize that North American Aerospace Defense (NORAD) is a 
bi-national arrangement that requires compatibility of systems and 
operations, we nevertheless expected that National Defence would 
have demonstrated that its requirements were being met, by clearly 
identifying what those requirements were and how the Battle Control 
System—Fixed (BCS-F) system responds to them. 


6.61 We examined whether the construction of the above-ground 
complex was supported by a sound rationale and business case. We 
found that the intended purpose for the building changed over the 
course of the project. It was originally planned to be a small operations 
centre, but it grew to be a larger facility. Part of the rationale for this 
construction was to save costs, but we found that the Department 
could not provide us with evidence to support how it had determined 
that there would be savings, how the changes to the building continued 
to support operational savings, or when savings would be realized. 


6.62 We are concerned that the Department is proceeding with the 
modernization of its air surveillance and control system without having 
clearly reviewed the Statement of Operational Requirement needed to 
help define the necessary characteristics of the system, and to guide 
the evaluation and performance standards of the system. The 
Department has not completed a business analysis that scrutinizes all 
costs as it moves forward. Nor has it prepared a business continuity 
plan. We are concerned by the fact that a statement, analysis, and plan 
are not yet in place, because the Department is forecasting further 
expenditures. These expenditures will be incurred for future 
components of the new air surveillance system. The Department has 
identified a project to examine this—the Canadian Air Defence Sector 


Upgrade Project. 
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About the Audit 


Objectives 


The objective of this audit was to assess National Defence’s development, implementation, and costs for 
modernizing the air surveillance and control system operated by the sector air operations centre under the 
North American Aerospace Defense(NORAD) agreement. Specifically, we assessed to what extent 
National Defence 


- complied with government policies, directives, and guidelines in managing project costs, timelines, 
and delivery; 


- complied with government’s policy and with its own policy on risk management in developing systems 


to replace the old AN/FYQ-93 (Q-93) system; 


» had a sound rationale for building an above-ground complex for housing a new air surveillance and 
control system to replace Q-93; and 


- analyzed options to ensure that the proposed replacements for the Q-93 system do in fact meet the 
NORAD and Canadian NORAD Region (CANR) operational requirements for the defence of North 


America. 


Scope and approach 


We conducted our audit at National Defence Headquarters, and we conducted field visits to the 
Cheyenne Mountain Operations Center (CMOC) in Colorado Springs, the CANR Headquarters in 
Winnipeg, and the Canadian Sector Air Operations Centre (SAOC) located at 22 Wing in North Bay. 


The audit team interviewed personnel responsible for the project at National Defence Headquarters. The 
team also interviewed personnel in the following groups: Assistant Deputy Minister (Information 
Management), Assistant Deputy Minister (Materiel), Assistant Deputy Minister (Policy), Assistant 
Deputy Minister (Infrastructure and Environment), Assistant Deputy Minister (Finance and Corporate 
Services), and Chief of the Air Staff. At each of these organizations, the audit team examined relevant files 
and documents. We interviewed personnel at the Cheyenne Mountain Operations Center in Colorado 
Springs, at the Canadian NORAD Region headquarters, and at the Sector Air Operations Centre. 


Expenditures were tracked using data from the Financial Management and Accounting System and from 
department reports. 


We attempted to reach a clear understanding of the Regional/Sector Air Operations Centre (R/SAOC) 
Modernization Project. We examined project management, costs, and timelines. 


All government funding approvals exclude the Goods and Services Tax (GST). 
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Criteria 
We expected that National Defence would do the following: 
- Comply with government policies, directives, and guidelines for contracting; 


- Have a completed option analysis and approved plans for addressing capability deficiencies as a way of 
meeting the requirements of the modernization project and the NORAD mission; 


» Demonstrate the savings realized by moving to the above-ground complex; and 


- Have analyses that support the decision to move the NORAD installation into the above-ground 
complex. 


Audit work completed 


Audit work for this chapter was substantially completed in January 2007. 


Audit team 


Assistant Auditor General: Hugh McRoberts 
Principal: Wendy Loschiuk 

Lead Director: Pierre Hamel 

Director: Tony Brigandi 


Director: Daniel Thompson 
Audit Project Leader: Craig Millar 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix List of recommendations 


The following is a list of recommendations found in Chapter 6. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Approval to modernize the NORAD system 


6.32 National Defence should update — Agree. In response to a meeting of the Senior Review Board on 


its analysis of the NORAD system 11 December 2006, the business case for the Region/Sector Air 
modernization project’s expected Operations Centre Modernization Project was updated on 
savings, and should determine when 15 February 2007. As a significant portion of the original 

such savings will be realized. business case was predicated on the possible relocation of the 
(6.8-6.3 1) operational elements, much of the infrastructure-related savings 


originally envisioned will not be realized. The current savings 
projections are approximately 18 percent of the original 

1996 business case estimate; further studies on the actual savings 
are ongoing. 


Although the underground complex has been declared surplus 
and is currently moving through the surplus property process, 
two additional challenges have impacted plans for closure of the 
underground complex. First, the underground complex has been 
identified as a heritage site. This will entail providing a minimal 
level of support to the underground complex until a permanent 
solution is found. Second, due to residual security concerns, a 
small number of systems have yet to be moved from the 
underground complex. This should be rectified over the next few 


months. 
Project management and oversight 
6.36 National Defence should ensure Agree. Canada and the US have continued to evolve the system 
that any further modernization and modernization requirements over time. With the full 
upgrade of Canada’s air surveillance participation of Canada in earlier NORAD requirements 
and control system is supported by a documents, our requirements were incorporated into the Battle 
current and clearly defined Statement Control System Operational Requirements Document and the 
of Operational Requirement. System Capabilities Document. In addition, as the Department 
(6.33-6.35) determines the need for follow-on effort, all capability 


deficiencies will be validated as part of the Statement of 
Requirements for the Canadian Air Defence Sector Upgrade 
Project. 
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6.40. The Department should conduct 
a threat and risk assessment for the air 
surveillance and control system, to 
determine the extent to which the 
Department will provide system 
back-up and redundancy, and to 
develop a business continuity plan as 
necessary. (6.37-6.39) 


Agree. The Department agrees that threat risk assessments and 
business continuity plans are important and necessary. For the 
Region/Sector Air Operations Centre Project, a threat risk 
assessment was prepared for the first system and revised for the 
MASE system. For the current BCS-F system, a threat risk 
assessment is being performed as part of the Department of 
Defence Information Technology Security Certification and 
Accreditation Process. Within this certification and 
accreditation process, back-ups, redundancy, and continuity will 
all be addressed. The requirement for site redundancy will be 
further examined under the Canadian Air Defence Sector 
Upgrade Project. 
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Management of Forensic Laboratory 
Services 


Royal Canadian Mounted Police 


All of the audit work in this chapter was conducted in accordance with the standards for assurance engagements set by the 
Canadian Institute of Chartered Accountants. While the Office adopts these standards as the minimum requirement for our audits, 
we also draw upon the standards and practices of other disciplines. 
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For the most part the FLS does not meet its own turnaround targets for completing service requests 
Average turnaround time has improved for service requests in all disciplines except Biology 

Only about half the clients we surveyed said that the service was timely 

The FLS is not prioritizing cases as described to the parliamentary standing committee 

Backlog is a major cause of longer turnaround times in Biology 

Additional resources have not made service more timely in Biology 


Quality of lab results 


The labs have a quality management system in place 
The quality assurance program failed to identify some quality issues 
The FLS did not follow adequate procedures for project planning and implementation 


Client consultation 


Clients lack opportunity to influence how the FLS operates 
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Management of Forensic Laboratory 
Services 


Royal Canadian Mounted Police 


Main Points 


The Forensic Laboratory Services (FLS) of the Royal Canadian 
Mounted Police (RCMP) provides forensic analysis services for 
criminal cases to Canada’s law enforcement community, which 
includes Canadian police agencies, Crown counsel, and other federal, 
provincial, and municipal agencies. It operates laboratories in six cities 
across Canada. 


In 2005, the House of Commons Standing Committee on Justice and 
Human Rights, Public Safety and Emergency Preparedness heard 
conflicting testimony on the performance of the Forensic Laboratory 
Services. The Committee subsequently asked the Auditor General of 
Canada to audit the performance of the FLS, and the status of DNA 
cases and service requests. We examined the timeliness of service 
delivery by the FLS, the quality management system, consultation with 
clients, and performance reporting to Parliament. We did not audit the 
quality of the forensic science and offer no opinion, positive or 
negative, on the validity of scientific analysis used. 


The RCMP’s Forensic Laboratory Services is the main provider of 
forensic analysis for most police agencies and courts in Canada. 
Forensic labs are an important element of the criminal justice system. 
Investigators and prosecutors rely on forensics to help identify or 
eliminate suspects and to provide evidence that can withstand scrutiny 
in court. Delays in obtaining results of forensic analysis can slow police 
investigations and leave criminals on the street to re-offend. 


- For the most part the FLS does not meet its own turnaround targets 
for completing service requests. Although it can process urgent 
service requests in less than 15 days, they account for only | percent 
of all service requests. In the remaining 99 percent categorized as 
routine, the FLS is unable for the most part to meet the 30-day target 
it has set for them. While average turnaround times have improved 
for all other types of analysis, for DNA analysis requests they have 
worsened—from 91 days in 2003—04 to 114 days in 2005-06— 
despite increased spending and additional staff. The backlog of DNA 


requests is a major contributor to the long turnaround times. 
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- Although the labs now have a national quality management system 


in place, in practice there are significant weaknesses in how the FLS 
defines, records, monitors, and resolves quality issues (situations 
where there is a concern that the validity of forensic work is 
unreliable for any reason), including those related to lab results. 
Furthermore, the national quality management system failed to 
identify problems with the new automated process for DNA analysis. 
At the conclusion of our audit work, the FLS quality management 
system was not functioning as designed and could not provide 
assurance of quality to senior management. 


The RCMP does not give clients—police forces and prosecutors— 
adequate opportunity to influence how the FLS operates. For 
example, clients told us that although the RCMP conducts client 
visits, it consults them very little on their needs regarding matters 
such as changes to the labs’ services, priorities, and service standards. 
Clients have little opportunity to negotiate turnaround times for 
service requests, and a recent change in policy gives them little say in 
the number of exhibits they are allowed to submit—no more than 
eight with each request. 


Although its new Laboratory Information Management System 
enables it to examine the performance of the FLS, the RCMP is not 
keeping its commitment to report to Parliament on performance; nor 
is it reporting to clients on FLS performance. 


The RCMP has responded. The RCMP agreed to our 


recommendations and is preparing action plans to address them. 
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Introduction 


7.1 Through its Forensic Laboratory Services (FLS), the Royal 
Canadian Mounted Police (RCMP) provides forensic analysis services 
to Canada’s law enforcement community, including police agencies, 
Crown counsel, and other federal, provincial, and municipal agencies. 


12 The FLS is a national laboratory service operating at six sites 
located in Halifax, Ottawa, Winnipeg, Regina, Edmonton, and 
Vancouver. It provides a range of forensic analysis services relating to 
criminal cases, chiefly in the disciplines of Biology, Toxicology, 
Firearms, Trace Evidence, and Counterfeit and Document 
Examinations (Exhibit 7.1). 


7.3 Other FLS services include a Break and Enter DNA processing 
unit; a national Breath-Test Program; an Integrated Ballistics 
Identification System to better link bullets to crime scenes; an 
explosives laboratory; a laboratory that analyzes unknown substances; 
and diamond profiling research, which will assist in combatting 
organized crime by determining the source and identity of a diamond. 
The FLS does not provide services related to fingerprints, computer 
analysis, or chemistry of illicit drugs. It also does not perform autopsies. 


7.4 Not every site provides a full range of services. Furthermore, the 
FLS does not serve the entire law enforcement community: the 
provinces of Quebec and Ontario operate their own forensic 
laboratories, and the Vancouver Police Department operates its own 
forensic firearms laboratory. 


Exhibit 7.1 FLS provides a range of forensic analysis services relating to criminal cases 


Discipline Type of service 


* Biology | DNA analysis 
a a eee —— 
¢ Toxicology Detection of drugs and alcohol in bodily 
fluid and tissue 
"» ‘Fireaims Firearms analysis to determine type of 


| weapon, and to compare firearms 


| cartridges with weapons 


Tool mark analysis to examine what tool 
may have been involved in a crime 


+ 


| Analysis of fibre, paint, glass, and 
explosives 


Trace evidence 


Analysis of counterfeit and questioned 
documents 


Bureau for Counterfeit and Document 
Examinations 
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A case may involve more than one service 
request, each concerning evidence submitted to 
a particular discipline for analysis. 


7.5 FLS headquarters in Ottawa provides overall management, with 
a national program manager responsible for each discipline. Other 
headquarters functions include operating the National DNA Data 
Bank and a quality assurance program. 


7.6 According to FLS records, expenditures during the 2005-06 
fiscal year amounted to $40.8 million ($25.2 million for salaries, 
$13.0 million for operations and maintenance, and $2.6 million for 
capital); its full-time staff numbered about 385 (Exhibit 7.2). The 
primary source of funding was the federal government, although the 
provinces and territories provided $5.7 million through Biology 
casework agreements. 


Exhibit 7.2 Expenditures and staff, 2005-06 fiscal year 


Actual $ 


“Euendiues. sta 1? taniftions) 
Case Receipt Unit | ; ; } 41 $2.5 
Biology (including Evidence Recovery Unit) 110 $9.4 
Toxicology if Sil $4.0 
Firearms 35 $2.9 
Trace Evidence Sil $2.7 
Bureau for Counterfeit and Document Examinations 20 | $U55 
National DNA Data Bank 28 $2.6 
Other (including administration, automated systems, 69 $15.4 
quality assurance, and operational support)! 
Total 385 $40.87 


! The provinces/territories covered $5.7 million of the expenditures. Some of this amount 
was used for the National DNA Data Bank. 


2 Excludes $1.4 million from the British Columbia government for a large forensic analysis. 
Because of rounding, individual amounts do not add up to the total. 


Source: Royal Canadian Mounted Police 


7.7. In the 2005-06 fiscal year the FLS received approximately 
8,300 cases, representing about 13,000 service requests. A single case 
may generate more than one service request, depending on the number 
of disciplines involved. Most service requests were for Toxicology, 
Biology, and Firearms analysis services (Exhibit 7.3). Amendments to 
the Criminal Code and the DNA Identification Act will broaden the 
types of offences entered into the National DNA Data Bank. Once the 
amendments come into force, FLS projections suggest that the number 
of cases received by the FLS could increase by more than 30 percent. 
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The RCMP categorizes a service request as 
urgent if it concerns an immediate threat to 
national security (for example, terrorist 
activities) or community safety (for example, 
unsolved serial crimes against persons). It 
categorizes all other requests as routine. 
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Exhibit 7.3 Toxicology, Biology, and Firearms accounted for most of the service requests received 


Toxicology 3,743 


Biology 3.362 
Firearms 3,032 
Bureau for Counterfeit and Document Examinations | 1,454 
No-suspect Break and Enter 696 
Trace Evidence 730 
Total 13,016 


Source: Royal Canadian Mounted Police 


71.8 Inthe early 2000s the RCMP restructured the FLS to improve 
efficiency. Previously, six full-service laboratories operated separately, 
each serving a limited geographic area. Now they function as a single 
consolidated organization with several service delivery sites across 
Canada. The new model has a number of notable features: 


¢ A Case Receipt Unit (CRU) at each laboratory provides a single 
point of contact with investigators, and can move cases between 
disciplines and laboratories. 


¢ An Evidence Recovery Unit (ERU) conducts a one-time search of 
exhibits to recover all potential evidentiary material. 


* A new case prioritizing system has been developed. The target is 
completion within 15 days for cases designated as urgent and 
30 days for cases designated as routine. 


* An off-the-shelf Laboratory Information Management System 
(LIMS) has been installed and customized. It ensures that staff 
members at different sites have access to the same case at the same 
time and that casework can be moved from one site to another. 


7.9 The consolidated structure aims to optimize casework activity by 
grouping scientists and technicians with similar training at the same 
site, and by reducing equipment and material costs. 


7.10 Under the new service delivery model, individual scientists do 


not follow cases from start to finish; instead, cases flow through 
specialized units. For instance, for a Biology service request, the CRU 
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receives exhibits, the ERU searches exhibits for evidence, the 
Analytical Unit analyzes what the ERU recovers, and the Reporting 
Unit interprets the findings and prepares a report (Exhibit 7.4). 


Exhibit 7.4 Under the new model for processing Biology cases, cases flow through specialized units 


Case Receipt ee Evidence oe Analytical Unit ae Reporting Unit 


Unit ee Recovery Unit 


| Receives exhibit | | Searches exhibit | | Analyzes DNA | | Interprets DNA J 
iS pees | profile and writes | 
report 


7.11 Other countries differ from the FLS in how they deliver services, 
particularly for Biology service requests. Some labs give one scientist 
responsibility for following a case through the entire process. Other 
labs function like the FLS, with units responsible for a specific aspect of 
the process. Still other labs adopt a team approach. In most of the labs 
we reviewed, the scientist responsible for reporting the findings plays a 
crucial role in overseeing the case from start to finish. At the FLS a 
case coordinator, who is not involved in analyzing the exhibits, is 
responsible for the administration and management of the case. 
Reporting officers are involved in scientific decisions if there are 
specific questions relating to the file or if samples are reworked. 


7.12 Forensic labs are an important element of the criminal justice 
system. Investigators rely on forensic science to help identify or 
eliminate suspects, and to link suspects to crime scenes. Prosecutors 
rely on forensic science because it can withstand scrutiny in court. 
Forensic evidence can also aid in the exoneration of persons previously 
convicted. 


7.13 DNA analysis is one of the most important tools available to 
forensic laboratories and has been used in Canadian courts since 1988. 
DNA evidence may be obtained from a small biological sample, such as 
a few drops of blood, semen, or saliva. Testing techniques may enable 
scientists to detect traces of blood even after clothing has been 
washed. However, because forensic samples may be contaminated, 
recovery of DNA is never certain. The RCMP’s National DNA Data 
Bank has been operational since 2000. It helps law enforcement 
agencies identify repeat offenders by comparing DNA from crime 
scenes with DNA profiles from offenders convicted of serious crimes. 
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Standing Committee on Justice and Human 
Rights, Public Safety and Emergency 
Preparedness—In April 2006, the House of 
Commons changed the committee's name to the 
Standing Committee on Justice and Human 
Rights. At the same time, the House established 
a new committee—the Standing Committee on 
Public Safety and National Security—to take on 
the balance of the former committee's mandate. 
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Previous audits and allegations 


7.14 


Our 1990 and 2000 audits of the RCMP included 


recommendations about the FLS (Appendix A). In our April 2000 
Report, Chapter 7, Services for Canada’s Law Enforcement 
Community, we noted that the level of service provided by the FLS did 
not meet clients’ needs. DNA analysis took too long and was restricted 
to only the most important cases, limiting this tool’s potential for 
enhancing public safety. 


7.15 


In 2005 the House of Commons Standing Committee on 


Justice and Human Rights, Public Safety and Emergency 

Preparedness heard testimony from two former RCMP staff members, 
casting doubt on statements made earlier by RCMP officials before the 
Committee. The two former members particularly disputed claims that 


7.16 


the RCMP Forensic Laboratory Services compared favourably 
with similar services around the world, 


there were no DNA case backlogs, 

DNA case response times were acceptable, 
service efficiency and effectiveness had improved, 
RCMP laboratories received adequate funding, 


the Balanced Scorecard (the RCMP’s internal performance 
reporting system) was being used to solve previously existing 
problems, and 


structural changes responded to the Auditor General’s 2000 
Report. 


The Committee asked the Auditor General to investigate the 


conflicting testimony presented to it on the performance of the RCMP 


Forensic Laboratory Services and on the status of DNA cases. 


Focus of the audit 


7.17 


The objectives of this audit were to examine whether 
the FLS is delivering timely forensic services, 
there is adequate quality control of lab results, 


clients have adequate opportunity to influence how the FLS 


operates, and 


the RCMP has kept its commitment to report on FLS 


performance to Parliament. 
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Timeliness of service — 


Turnaround target—The number of days within 
which a service request should be completed. 


7.18 The scope of the audit is limited to the RCMP’s Forensic 
Laboratory Services. We did not examine other RCMP functions or 
services, or the National DNA Data Bank. Nor did we audit the 
quality of the forensic science used by the FLS, and we offer no 
opinion, positive or negative, on the validity of scientific analysis used. 


7.19 More details on the audit objectives, scope, approach, and 
criteria are in About the Audit at the end of this chapter. 


Observations and Recommendations 


7.20 Forensic analysis is a highly valuable tool for police 
investigations, and delays in obtaining results can have significant 
impact. Delays may slow investigations and hamper the police’s ability 
to eliminate suspects and make arrests quickly; the results include 
increased costs and wasted resources. More important, delays endanger 
public safety by giving criminals more time on the street to re-offend. 


7.21. In our 2000 audit we noted numerous complaints about delays in 
DNA analysis by the Forensic Laboratory Services (FLS) and 
recommended better prioritizing so that more urgent requests would be 
handled first. We reported that on average DNA analysis took 101 days 
to complete at the end of 1999 (Vancouver and Ottawa labs averaged 
171 days) and that there was a substantial backlog of cases not 
completed. Former RCMP employees have disputed recent claims to 
Parliament by the RCMP that turnaround times for DNA testing were 
improving and that the backlog was decreasing. Our objective was to 
determine whether the FLS is now providing timely service and whether 
there is a backlog. We examined the FLS from the 2003—04 fiscal year, 
when the new Laboratory Information Management System was 
implemented, through the 2005-06 fiscal year. 


For the most part the FLS does not meet its own turnaround targets for completing 
service requests 


7.22 The FLS introduced turnaround targets for service requests in 
the early 2000s. In 2000 the House of Commons Standing Committee 
on Public Accounts recommended that the RCMP “develop and 
implement a series of performance indicators and standards for each of 
the services provided by its forensic laboratories by the end of fiscal 
year 2000-01.” The government responded that the RCMP had 
developed a two-tier priority system to replace the existing four tiers, 
with turnaround targets of 15 days for urgent requests and 30 days for 
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routine ones. In 2000 the RCMP created a performance standard 
formula with the same turnaround targets, including waiting time. The 
RCMP also called for targets to be validated, and it estimated that it 
would have sufficient data to undertake the validation by April 2002. 
In fact, no validation was ever conducted. 


7.23 At the time of our audit, FLS policy included the 15- and 30-day 
turnaround targets, and the FLS reported on the extent to which it 
met these targets in the Balanced Scorecard. Since the FLS itself 
established the turnaround targets, we determined that it was 
appropriate to expect this standard to be met. 


7.24 In the 2005-06 fiscal year the FLS met the 15-day turnaround 
target for handling almost all urgent requests—132 out of 134. But 
these represented only | percent of requests completed. The remaining 
99 percent were categorized as routine, and in most instances the FLS 
did not meet its 30-day turnaround target for them. It met the target 
for only 10 percent of routine Biology service requests, 55 percent of 
Toxicology requests, 34 percent of Trace Evidence requests, 

and 23 percent of requests in both Firearms and Counterfeits/ 
Documents. 


7.25 A number of sources indicate that the turnaround targets 
established by the FLS reflect operational requirements. For instance, 
in his 1996 inquiry into police investigation of the Bernardo case, 
Justice Archie Campbell recommended a 30-day turnaround time for 
DNA analysis. The RCMP performance standard formula created 

in 2000 set 15- and 30-day turnaround targets. In 2001, clients called 
on the FLS to establish a standard of 5 days or less for urgent requests, 
and 30 days or less for routine ones. In a survey we conducted for our 
audit, clients said that 15- and 30-day targets were acceptable. There is 
no generally accepted international standard, but the turnaround 
targets are similar to those set by some other labs. In the US state of 
Georgia, for example, the goal is to handle priority requests 

within 20 days and regular requests within 30 days. In Sweden the 
target is 20 days for all requests. Other labs have the goal of 
completing a certain percentage of cases within a set period; for 
example, the Ontario Centre of Forensic Sciences has a goal of 
completing 80 percent of cases within 90 days. 


7.26 In addition to the corporate target of 30 days, the FLS uses 
Expected Diary Dates (EDDs) to provide realistic information to clients 
about the amount of time required to complete a routine service 
request. The EDD for Biology is currently 180 days. Management 
expects EDDs to be met 100 percent of the time. The proportion of 
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EDDs currently being met has risen from previous years but it has 

not reached 100 percent. Only 74 percent of Biology requests were 
completed within the 180-day EDD during the 2005-06 fiscal year. 
Other disciplines had higher completion rates, ranging from 80 to 

98 percent of their EDDs. However, the FLS has lengthened the target 
times in recent years. For example, the EDD for Biology increased from 
70 to 180 calendar days over the last three years. Although the 
corporate target of 30 days for routine requests may not currently be 
realistic for the FLS, an EDD that can be changed to ensure it is met is 
not a good performance indicator. It is important that the FLS develop 
appropriate targets for measuring performance. 


7.21 In January 2006 the FLS began developing the Priority Rating of 
Operational Files (PROOF) system—a new prioritizing system that 
ranks operational files from least to most important. PROOF keeps the 
two-tier system of “urgent” and “routine,” but it further divides routine 
cases into streams A, B, and C according to the type of offence and 
other criteria. PROOF is currently being implemented in a pilot 
project, and turnaround targets have not yet been established for the 
different streams. That is, a similar time frame applies for handling 
lesser offences (stream C) and more serious ones (streams A and B). 
The PROOF system has not been operating long enough for us to 
comment on the results. 


7.28 Recommendation. The RCMP should ensure that the Forensic 
Laboratory Services’ prioritization system and turnaround targets meet 
the operational needs of clients. Turnaround targets should be 
implemented before service requests increase, as they are expected to 
do once amendments to the Criminal Code and the DNA Identification 
Act come into force. The targets should be used to measure and report 
FLS performance. 


The RCMP’s response. The RCMP agrees that response times must 
be reduced. As the report indicates, significant reductions have been 
achieved in all disciplines except Biology (DNA). 


The RCMP agrees that client consultation is important in ensuring 
that response times meet client needs. The RCMP will continue to 
engage its clients to establish reasonable turnaround targets. These 
targets will form the basis of performance reporting. 


The FLS acknowledges the significant additional demands on Biology 
services that the proposed changes to the Criminal Code and the DNA 
Identification Act will create. The FLS has identified its requirements to 
meet the proposed changes. 
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Turnaround time—The time that elapses from 
the arrival of the evidence at the FLS until the 
FLS sends a final report. 
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Average turnaround time has improved for service requests in all disciplines 
except Biology 


7.29 Recent years have seen a decrease in the average number of days 
required to complete a service request in most disciplines (Exhibit 7.5). 
When urgent and routine service requests are combined, the average 
turnaround time in Toxicology declined from 52 days in the 2003-04 
fiscal year to 46 days in 2005-06. Over the same period it fell from 
147 to 86 days in Firearms, 82 to 59 days in Trace Evidence, and 131 to 
60 days in Counterfeits/Documents. With the exception of Firearms, 
these program areas also experienced a decrease in the number of 
service requests received over the same time frame. According to 

FLS officials, this may have given staff time to complete requests 

more expeditiously. 


Exhibit 7.5 The average turnaround time has improved for all disciplines except Biology 


Number of days 
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Biology Toxicology Firearms Trace Evidence Counterfeits/ 
Documents 


a 2004-05 FB 2005-06 


Number of completed service requests 


Discipline 


Biology | 49038 
“Toxicology eh 3,945 3,856 3,684 
Firearms | 23935 | 2,408 | 2,682 
Trace Evidence 852 | 688 772 
Counterfeits/Documents | 1,755 ; 2,239 1,399 


Source: Royal Canadian Mounted Police 
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7.30 The average overall turnaround time in Biology increased from 
91 days in the 2003-04 fiscal year to 114 days in 2005-06. This includes 
requests outsourced to a private laboratory; excluding them, the average 
turnaround time in the 2005-06 fiscal year was 118 days (compared with 
94 days in 2003-04). Since the FLS categorizes service requests as either 
urgent or routine, we examined turnaround times for both priority levels. 
An urgent Biology service request is completed in 11 days on average. 
However, urgent requests account for only about 3 percent of completed 
Biology service requests. The remaining 97 percent of requests are 
routine and their average turnaround time is 116 days. 


7.31 In statements to Parliament, the RCMP noted that its forensic 
services are “as good as, if not better than, anywhere else in the world” 
in terms of response time for major crimes. At the time of the audit, 
however, the FLS was not able to provide us with much information on 
how it compares with other labs. We therefore contacted other labs in 
Canada and elsewhere (Exhibit 7.6). Our information on those facilities 
is based on official reports or was provided to us during site visits and 
later confirmed. It is unaudited. Although some labs report shorter 
turnaround times than the FLS, few labs complete service requests 
within 30 days, and some labs reported longer turnaround times than 
the FLS. The United Kingdom’s Forensic Science Service, a private 
organization, appears to have the shortest turnaround time, with an 
average of 7 days in the 2004—05 fiscal year fora DNA crime scene 
request. For other labs we visited, turnaround times range from a 
median of 28 days at Sweden's National Laboratory of Forensic Science 
(excluding break and enter samples, which are generally completed 
more quickly) to more than 100 days in some labs in the United States. 
In Canada, the Ontario Centre of Forensic Sciences reports an average 
turnaround time of 96 days (excluding break and enter samples). 


Only about half the clients we surveyed said that the service was timely 


7.32 According to RCMP officials, efficiency and effectiveness of 
casework processing are measured against client needs and satisfaction 
levels, as determined by quality of service questionnaires and client 
visits. The FLS sends out questionnaires with each report it prepares, 
and it analyzes those returned to measure client satisfaction. 
According to returned questionnaires in the 2005-06 fiscal year, 

87 percent of clients agreed or strongly agreed that the service was 
timely. However, these questionnaires are not anonymous and the 
response rate was only 33 percent. The FLS has not studied whether 
clients who do not respond to these questionnaires are more likely to 
have negative opinions. 
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SR te ee 
Exhibit 7.6 We compared RCMP FLS to other Biology units! 


Sage Expenditures Service requests | Average turnaround 
Pit | (millions of CANS)? completed yearly time (days) Comments 
RCMP FLS $9.4 3,223 114 * In addition, completed 
592 no-suspect break 
and enter service 
requests 
Ontario Centre of $5.9 75 3,682 96 | « Expenditures and staff 
Forensic Sciences cover work on break 
and enter service 
requests and some 
research and 
development 
* In addition, completed 
1,402 no-suspect 
break and enter 
service requests 
Laboratoire de $4.8 38 4,319 — ¢ Includes break and 
sciences judiciaires et enter service requests 
de médecine légale, 
Quebec 
Georgia Bureau of $2.0 16 5,03 80-105 ¢ Includes break and 
Investigation, Division (estimate) enter service requests 
f Fi i i 
Apa cape ¢ Does not include about 
3,000 outsourced 
service requests 
* Service requests may 
be split, which could 
affect average 
turnaround time 
Florida Department of | $8.8 108 | 6,621 188 ¢ Includes break and 
Law Enforcement, enter service requests 
Crime Laboratory (US) 1 
1 ms = = = ——— —= 
UK Forensic Science — — _ Vf ¢ Private lab model 
Service (crime scene 
exhibits only) 

DJ ee = 4 = = 
National Laboratory of $5.1 65 Approx. 28 ¢ Expenditures and staff 
Forensic Science, 3,000 (median) | cover work on the DNA 
Sweden | database and break 

and enter service 
requests 
¢ In addition, completed 
10,000 break and 
enter service requests 


! The information provided by the UK is based on a published 2004-05 annual report; other information was provided to us directly by the 
labs. It is unaudited. Despite some organizational or policy differences, in general labs have indicated that the information is based on 
definitions similar to what the FLS uses. The time frame used differs slightly across labs, but basically covers 2004—05 or 2005-06. 


Unless noted, the information excludes processing of requests by the DNA data bank. 
* Converted to Canadian dollars using Bank of Canada annual average exchange rates. 


- Comparable information is not currently available. 


Report of the Auditor General of Canada —May 2007 Chapter 7 13 


MANAGEMENT OF FORENSIC LABORATORY SERVICES—ROYAL CANADIAN MOUNTED POLICE 


Primary offences are the most serious of 
crimes. They include murder, attempted murder, 
sexual assault, and assault. 


7.33 We conducted a client survey with investigators responsible for 
113 randomly selected cases with at least one primary offence, and 
another more general survey of 135 clients, with response rates of 

98 and 91 percent respectively (see About the Audit). We asked 
clients about the timeliness of service for specific service requests, as 
well as the timeliness of service in general. In client interviews about 
service in general, around one third of respondents said the response 
time was good or very good for urgent requests, and one quarter said it 
was good or very good for routine requests. In discussing specific cases, 
half (53 percent) of those interviewed said that the service was timely. 
About one third (30 percent) said the response time was fair, and 

18 percent said it was poor or very poor. When asked what would have 
been an appropriate turnaround time for the specific service request 
they were asked about, 68 percent said 30 days or less. These results 
differ substantially from those of the FLS questionnaires, indicating the 
need for additional measures of client satisfaction by the FLS. 


7.34 Recommendation. The RCMP should develop measures of 
service efficiency and effectiveness in consultation with clients. In 
addition to existing tools for measuring client satisfaction, it should use 
client surveys conducted by an independent, external organization. 


The RCMP’s response. The RCMP agrees that measurement of 
efficiency and effectiveness, both internally and from client feedback, 
is important. The FLS currently uses a Quality of Service 
Questionnaire, which it encourages clients to complete. The RCMP 
agrees that use of an independent third party to develop and receive 
feedback may improve response rates and reduce the likelihood of bias. 
It will further explore this option. 


The FLS is not prioritizing cases as described to the parliamentary standing 
committee 


7.35 In their 2005 testimony before the House of Commons Standing 
Committee on Justice and Human Rights, Public Safety and 
Emergency Preparedness, RCMP officials said that “murder cases, 
cases with violence, violent assault cases . . . would definitely go right 
into the priority queue and be handled right away.” In fact, the FLS 
does not prioritize cases in this way. At the time of the testimony an 
urgent service request was defined as concerning only an immediate 
threat to national security (for example, terrorist activities) or 
community safety (for example, unsolved serial crimes against 
persons). Other requests could be designated as urgent under 
extraordinary circumstances—for example, if the ability to recover 
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information from the material declined rapidly, or a court order 
required a time frame of less than 30 days. 


1.36 There were 63 service requests categorized as urgent during 

the 2004-05 fiscal year and 134 in 2005-06 (about two thirds of them 
Biology requests). In each of those years the urgent requests amounted 
to | percent or fewer of all requests. In contrast, there were 980 urgent 
requests amounting to 8 percent of the total in the 2003-04 fiscal year, 
when the definition of “urgent” was broader. Approximately 38 percent 
of all service requests in 2005-06 were related to violent crimes, 
including murder, sexual assault, robbery, abduction, and assault. A 
further 10 percent were related to property crimes (such as break and 
enter, and theft) and 52 percent were related to “other” Criminal Code 
offences (such as motor vehicle or weapons offences and fraud). Since 
almost 5,000 of completed service requests were related to violent 
offences and 134 service requests were categorized as urgent, it follows 
that most violent cases in fiscal year 2005-06 were categorized as 
routine. Officials told us that the FLS lacks the capacity to treat large 
numbers of service requests as urgent. 


7.37 To evaluate RCMP claims that serious offences such as murder 
and violent assault are handled more rapidly than requests involving 
less serious crimes, we calculated turnaround times for different 
offence categories. In the 2005-06 fiscal year, service requests for 
violent offences were not handled more rapidly than those involving 
less serious crimes. For instance, the average turnaround time was 

91 days for service requests related to homicide and 109 days for those 
related to sexual assault. For service requests related to break and enter 
the average turnaround time was 87 days, and for those related to 
motor vehicle offences it was 37 days. 


7.38 RCMP officials have noted that they strive to meet the court date 
set for a case. Aside from the need to meet a court date, there are other 
reasons for providing timely service. For instance, the results of DNA 
analysis can help eliminate suspects, allowing investigators to better 
focus their work and save time and resources. DNA evidence is often 
required to lay a charge; timely analysis can help ensure that a suspect 
is not on the street with the opportunity to re-offend. In addition, 
forensic evidence can alleviate victims’ anxiety, may be needed at 


preliminary hearings, and may exonerate an innocent person. 


7.39 The turnaround time for analysis of DNA warrants illustrates the 
significance of timely results. In many criminal investigations, police 
officers require an initial screening of DNA from the crime scene before 


they can obtain a court order for the collection of a suspect’s DNA. 
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Backlog—Requests for service not completed 
within the approved performance standard 
response time for the assigned priority—in this 
case the 30-day routine standard developed by 
the FLS. 


Queued cases involve secondary offences (for 
example, robbery or break and enter) and are 
beyond the quota of cases that the FLS is 
expected to complete under agreements with 
provinces/territories. 


The two steps open the possibility for two delays. First is the wait for the 
initial screening of DNA from the crime scene before obtaining a DNA 
warrant; second is the wait to obtain a suspect’s DNA profile and check 
whether it matches the crime scene sample. Until October 2006, the 
FLS had no mechanism in place for linking and expediting the two 
steps. Officials told us that the FLS can now link DNA warrant samples 
with initial screenings and give them higher priority. 


Backlog is a major cause of longer turnaround times in Biology 


7.40 We looked for delays in particular parts of the process for Biology 
service requests. FLS officials have said that cases are not opened as 
promptly as they once were, lengthening the turnaround time. We 
could not use the Laboratory Information Management System 
(LIMS) to calculate processing and waiting times because the FLS 
decided not to include information on waiting times in that system. 
The FLS told us that no one has measured the exact time required for 
particular tasks, but it estimates that processing time for a Biology 
service request ranges from 11 to 21 calendar days. This includes 2 to 
6 days for evidence recovery, 5 to 6 days for analysis, 3 to 5 days for 
reporting, and | to 4 days for technical and administrative reviews. If 
we subtract processing time from the average turnaround time of 

114 days, we obtain an estimate for the average waiting time of 


between 93 and 103 days. 


7.41. In 2001 RCMP officials reported that turnaround times for DNA 
testing were shortening and the backlog was decreasing. In 2004 they 
told the Standing Committee on Justice and Human Rights, Public 
Safety and Emergency Preparedness that the system had no backlog, 
only “cases in process.” 


7.42 The FLS regularly monitors requests that are not completed. 
According to FLS records, as of March 2006 there were 2,017 requests 
across all disciplines that were unfinished 30 days after receipt, a 
decrease from 2,518 in April 2003. However, during the same period 
the backlog in Biology grew from 663 to 873 service requests 

(Exhibit 7.7). Furthermore, at the end of March 2006, the Biology unit 
had a queue of 760 cases (which would translate to at least as many 
service requests) above the number agreed to in provincial/territorial 
agreements. These queued cases had not yet been accepted by the 
FLS. With the current performance standard of 2 requests per week for 
Biology reporting officers, additional staff may be required to clear 


the backlog. 
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Exhibit 7.7 The backlog in Biology has increased 
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Source: Royal Canadian Mounted Police 


7.43 Recommendation. The RCMP should develop mechanisms for 
identifying bottlenecks in the process and should determine the 
systems, procedures, and resources required to eliminate the backlog. 


The RCMP’s response. The RCMP agrees that identifying 
bottlenecks in processes is important, and will undertake workflow 
analyses to determine further efficiencies. 


Additional resources have not made service more timely in Biology 


7.44 According to FLS officials, a review completed in 2003 sought to 
ensure that programs were aligned with the FLS mandate. Officials 
said that the program review would yield annual ongoing savings of 
$2 million as a result of the consolidation of equipment and staff. 
Savings generated by the efficiencies achieved were to be re-invested 
in high-priority areas, such as DNA analysis and the creation of case 


receipt units and evidence recovery units. 


7.45 The RCMP has been unable to provide evidence of the program 
review but it appears to have adjusted funding levels in line with 
demand. In the 2005-06 fiscal year Toxicology, Trace Evidence, and 
Counterfeits/Documents received less funding than in 2003—04; this 
paralleled a decline in the number of service requests they received. 
Funding rose for Biology and Firearms, the two disciplines with an 
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increased number of service requests. Biology, for example, 
experienced a 5 percent increase in the number of service requests 
received between fiscal years 2003-04 and 2005-06. During the same 
time expenditures adjusted for inflation increased by at least 

30 percent (from $6.9 million to at least $9 million) and staff 

by 29 percent (from 85 to 110). 


7.46 According to FLS officials, however, it takes up to two years to 
prepare a scientist to conduct forensic analysis, and there are indirect 
costs of using in-house personnel for training. FLS officials could 
provide only limited data on the number of employees trained and 
available for work. On the basis of the data provided to us, if we exclude 
Biology staff in training, on leave, or not working in the unit, as well as 
indirect staff time used to train new personnel, we estimate that there 
was an increase of about 30 percent in staff devoted to casework from 
May 2004 to March 2006. FLS officials told us that most new staff 
members were assigned to the ERU, and the bottleneck is currently in 
the Analysis Unit. They told us that for turnaround times to decrease, 
overall demand for services must remain stable over time and the 
system for processing forensic DNA analysis must stabilize and mature. 


7.41 While resources for Biology have increased, the number of 
service requests completed each year has remained relatively stable: 
3,335 requests were completed in the 2003-04 fiscal year and 3,223 
in 2005-06. Similarly, according to FLS documents, there has been no 
major change in the number of exhibits per service request; these 
averaged 9.1 in the 2003-04 fiscal year and 9.9 in 2005-06. 
Accordingly, we would expect to see an improvement in the 
turnaround time for Biology service requests. Instead, the average 
turnaround time increased by 25 percent between fiscal years 2003-04 
and 2005-06 (from 91 to 114 days). The Biology unit was able to 


complete about 13 service requests per working day. 


7.48 We looked at the possibility that FLS involvement in a British 
Columbia serial murder case might have adversely affected turnaround 
times for other Biology cases. The FLS first became involved in the 
case in February 2002. During a five-year period it analyzed more 
than 220,000 DNA samples, by far the largest number analyzed for a 
single case in the history of the FLS. Between fiscal years 2002-03 
and 2005-06, the FLS received $8.2 million from the province for the 


forensic services it provided for the case. 


7.49 FLS officials told us that existing staff worked on the case until 
April 2004, when additional employees were hired. The officials also 
noted that the case had at least an indirect negative impact on other 
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operational casework. For instance, the FLS had to allot resources for 
hiring and training new staff; and the case required the attention of 
senior staff members, lessening the opportunity for them to address 
issues of turnaround time on other cases. Aside from the indirect impact, 
the FLS has not been able to quantify the direct impact on operational 
casework. As we have noted, during this period the FLS saw an increase 
in the number of staff available to handle the normal Biology workload. 


7.50 Recommendation. The RCMP should conduct a review of the 
Forensic Laboratory Services to examine internal efficiencies, perform 
a cost/benefit analysis of various services, and examine the need for 
additional resources. 


The RCMP’s response. As per recommendation 7.43, the RCMP 
agrees that there should be cost/benefits analyses for all services and 
will undertake a workflow analysis to determine if there are further 
operational efficiencies to be gained. 


1.51 Recommendation. The RCMP should develop a capability for 
management of the Forensic Laboratory Services to analyze capacity 
and efficiency (including comparing performance with that of other 

forensic labs) in order to handle future demands. 


The RCMP’s response. The RCMP agrees that benchmarking is an 
important tool to measure performance and analyze capacity to meet 
future demands. The FLS will strive to balance capacity with 
performance by developing individual and unit performance metrics, 
and ensure that they are measurable and reportable. 


The labs have a quality management system in place 


7.52 It is crucial that forensic work be of high quality. If it contains 
errors, guilty individuals could go free or innocent individuals could be 
wrongly convicted. Weak quality control in forensic laboratories can 
cast serious doubt on the integrity of the criminal justice system. 


7.53 Our 2000 audit found the FLS quality assurance process to be 
weak. The accreditation process was two years behind schedule, and 
the proficiency testing program was not adequately implemented or 
monitored. We recommended that the RCMP ensure quality in its 
laboratories by timely accreditation, documentation of methodology, 
and full implementation of the proficiency testing program. 


7.54 Since that time the FLS has made progress in remedying the 
weaknesses identified. It has created a national quality assurance 
program, with a National Quality Manager and Local Quality 
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Managers at each site. It has also established national policies and 
procedures regarding quality. The Standards Council of Canada has 
now accredited all FLS sites. 


7.55 The FLS has implemented another important aspect of quality 
assurance: a proficiency testing program. According to FLS policy, 
each forensic scientist must perform at least one proficiency test per 
year in his or her specialization, and at least one proficiency test every 
two years in each major area of testing within the program. Proficiency 
tests are the responsibility of the quality assurance program and are 
monitored through managerial reviews. Failure on a proficiency test 
results in an identification of a quality issue. 


7.56 According to FLS annual reports, success rates on proficiency 
tests are high—between 98 and 100 percent in calendar years 2001 
through 2005. The FLS proficiency testing program is similar to that of 
other labs, although some have more extensive testing in place (for 
example, blind testing or retests of performed analyses). The FLS 
primarily uses open testing and regards blind testing as unfeasible. 


7.57 Finally, in our review of quality of service questionnaires and in 
client interviews, we found that clients were generally pleased with the 
quality of service and particularly with staff. Eighty-nine percent of 
clients interviewed said that the quality of service provided by the FLS 
was good or very good. 


The quality assurance program failed to identify some quality issues 


7.58 FLS policy defines a quality issue as “any situation in which there 
is a reasonable concern or suspicion that the validity of forensic work 
undertaken in FLS in a particular instance is unreliable for any reason 
whatsoever. This includes situations in which the quality of service is 
suspected to be substandard.” 


7.59 Quality issues may result from clients’ complaints, proficiency 
tests, reviews, audits, or other sources. The National Quality Manager 
is responsible for the documentation, investigation, and adjudication 
of quality issues. FLS policy requires all staff to report suspicious 
situations to a Local Quality Manager. The Local Quality Manager will 
immediately notify the National Quality Manager, who in turn notifies 
the program manager. Consulting as necessary, the National Quality 
Manager and Local Quality Manager determine whether a suspicious 
situation should be recorded as a quality issue. According to FLS’s 
Standard Operating Procedure, if there is any doubt, the situation 
should be called a quality issue. 
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7.60 The National Quality Manager has the overall responsibility for 
investigating, identifying corrective and preventive actions, and 
preparing a written report; if the issue is local, however, the Local 
Quality Manager will conduct the investigation. The National Quality 
Manager is also responsible for maintaining a record of all quality 
issues, the results of investigations concerning them, and the actions 
taken to correct them. 


7.61 We found that in practice some issues meeting the FLS definition 
of a quality issue were not recorded as such. In 2005 the FLS identified 
11 quality issues. However, some nonconformities identified from 
internal reviews and other sources were not dealt with according to 
policy under the national quality management system. According to 
FLS officials, individual program assurance processes dealt with these 
issues. But in these instances, the National Quality Manager may not 
have been aware of a potential quality issue, the issue may not have 
been dealt with in a consistent manner, and some quality issues would 
not have been identified as such. 


7.62 FLS officials told us they lack a workable definition of a “quality 
issue” that would allow staff to distinguish between anomalous results 
needing investigation and simple lab errors. Consequently, senior 
management cannot determine whether the quality management 
system is operating as intended or whether any of the nonconformities 
had an impact on services provided. 


7.63 The most significant failure of the quality management system 
identified in the audit concerned the new automated process for DNA 
analysis, which was introduced in operational casework in 

September 2005. The automated process uses robots at various stages 
of analysis to increase the number of DNA samples that can be 
processed at one time. The RCMP’s Departmental Performance 
Report for the 2005-06 fiscal year projected that automation would 
“provide a two- to three-fold increase in casework capacity while 


. . . . kd . a os te 
improving the timeliness of reporting results. 


7.64 Shortly after implementation of the automated process, FLS 
scientists involved in casework began raising quality concerns in 

email messages to management and in meetings (Exhibit 7.8). Similar 
concerns were identified in a review conducted by a Local Quality 
Manager in November 2005. Late in that month the automated process 
was suspended to examine inconsistencies in results. Management 
concluded that the standard used for quantification was inaccurate, 
yielding weak DNA profiles. Management reinstated the automated 
process on 30 January 2006, for use in operational casework. About a 
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month later, FLS scientists again began expressing concern about the 
quality of results obtained from the automated process. 


7.65 In May 2006, FLS officials asked Biology staff to provide a list of 
case samples for which the automated process did not produce the 
expected results. As of 28 June 2006, 116 cases with a total of 

416 samples were identified, representing about 27 percent of cases 
processed since the implementation of the automated process. FLS 
officials noted that some samples identified did not conform to the 
criteria they had suggested. At the same time, it is possible that the list 
does not contain all samples with questionable results. We found 
instances in which the validity and consistency of the results were 


Exhibit 7.8 Problems with the automated DNA process were noted soon after implementation 


Year Date ve Event 


2005 26 September Analysis of operational casework using the new 
automated process for DNA analysis began. 


27 October Problems with the automated process were first noted. 


8 November An internal review identified concerns about the quality 
of the automated process. 


30 November Casework using the automated process was suspended 
pending a review of inconsistent results. 


2006 30 January Casework using the automated process restarted. 


March Starting in March and continuing throughout the year, 
staff members raised concerns about the automated 
process (for example, low DNA yields) and referred to 
quality assurance and quality control issues in meetings 
and email. 


10 May Management requested information on case samples 
that did not provide expected results. 


27 June The Acting National Quality Manager announced a 
review of cases for which the automated process had 
yielded questionable results. 


28 June Spreadsheet of 416 problem samples identified. 


18 July Troubleshooting progress report on the automated 
process: problem still unresolved. 


10 August Email indicating that certain sample types would now be 
processed manually. 


30 October Quality issue identified: using Hemastix on possible 
blood sample could interfere with any subsequent 
analysis using the automated process. (See definition of 
Hemastix on page 23.) 
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questioned by FLS scientific staff involved in operational casework, as 
well as by police investigators (Exhibit 7.9). FLS scientists were not 
concerned about errors in DNA profiles, once generated. Instead they 
were concerned about findings of no or insufficient DNA when they 
would have expected to find some. 


7.66 FLS officials have noted that there is no guarantee of obtaining 
DNA typing results from casework samples and that the previously 
used manual extraction process could give similar results. Furthermore, 
initial analyses indicate that the automated and manual processes yield 
similar overall success rates. However, in examining sample types (for 
example, where there was an indication of blood or where blood was 
identified), the FLS found that the success rate of the automated 
process in obtaining reportable DNA profiles was lower than the 
success rate of the manual extraction method. The FLS also found that 
when the automated process was used, approximately 3 percent of 
known samples did not yield sufficient DNA for profile development; 
when the manual process was used, the percentage was zero during the 
same period. The FLS did not identify a quality issue until 

October 2006, more than a year after concerns were first raised by 
staff. The quality issue was identified because of indications that the 
use of a Hemastix test strip on possible blood could interfere with 
analysis using the automated process. 


Exhibit 7.9 Examples where results from automated process were questioned 


¢ The FLS received exhibits from a murder investigation. According to the 
investigator, two exhibits had more than 100 spots of blood each. The FLS 
examined the samples using the automated process. Six months later it reported 
finding no DNA for analysis. The investigator asked for retesting because the case 
relied on DNA evidence and the exhibits clearly had blood on them. The FLS 
re-examined the exhibits with the manual method previously in use for analyzing 
DNA; it found positive DNA profiles linking the suspect, victim, and crime scene. 
Eleven months had elapsed from the original submission of exhibits until the final 
results were obtained. Although DNA was eventually found for the exhibit, this 
example illustrates concerns raised about finding no or insufficient DNA, and the 
added time involved in analysis. 


The FLS received exhibits from a murder investigation. One sample drawn from a 
suspect was to be used as a known sample. After the sample was examined, the 
analysis was rerun to confirm the profile. The second profile was not consistent with 
the first. The scientist responsible for the analysis raised the lack of reliable results 
as a quality issue. This issue was never formally identified as a quality issue or 
addressed through the quality management system. This example illustrates 
concerns raised about the consistency of results, and indicates that quality 
concerns were not reflected in the quality management system. 
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7.67 FLS officials noted that the November 2005 review conducted 
by the Local Quality Manager and subsequent actions were not 
brought to their attention until revealed by our audit in June 2006. 
Furthermore, although quality concerns were raised in internal 
meetings and through emails, and although the automated process was 
suspended for two months after concerns were raised, no quality issue 
was recorded until more than a year later, in October 2006. Since the 
concerns about the automated process fit the FLS definition of a 
quality issue, they should have been identified as such, with 
appropriate follow-up actions tracked and recorded. 


7.68 Recommendation. The RCMP should take measures to ensure 
identification of all quality issues. The RCMP should ensure that 
quality issues, once identified, are systematically tracked and resolved, 
and made available in a consolidated form, and that actions are 
communicated to senior management. 


The RCMP’s response. The FLS agrees that it will put into place 
managerial mechanisms to ensure that quality issues are defined 
clearly, are recorded and tracked in a systematic manner in its 
Quality Management System, are available in a consolidated format, 
and that all actions are communicated to senior management. 


The FLS did not follow adequate procedures for project planning 
and implementation 


7.69 The audit identified some additional concerns about the 
validation and implementation of the automated process. The process 
was validated for use in operational casework according to the 
guidelines of the Scientific Working Group on DNA Analysis Methods 
(SWGDAM). This working group is organized by the United States 
Federal Bureau of Investigation and includes representation from 
federal, state, and local forensic laboratories in the United States. The 
FLS is a member of SWGDAM. SWGDAM serves as a forum to share 
and evaluate forensic biology methods, protocols, training, and 
research to enhance forensic biology services. It also makes 
recommendations on quality assurance standards for forensic DNA 
testing laboratories. 


7.70 According to the 2003 SWGDAM guidelines, developmental 
validation (demonstration of the accuracy, precision, and 
reproducibility of a procedure) must precede the use of a novel 
methodology for forensic DNA analysis. Under the guidelines, a 
peer-reviewed publication of the underlying scientific principles of a 
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technology is required, and a peer-reviewed publication of the results 
of developmental validation is encouraged. The new procedure may be 
implemented without a peer-reviewed publication if the results have 
been disseminated in multiple ways to the scientific community for 
review and evaluation. Following the developmental validation, an 
internal validation is required to demonstrate the reliability and 
limitations of the procedure at each site where it is being used. 


771 According to FLS officials, the developmental and internal 
validations of the automated process were planned along SWGDAM 
guidelines, which cover the type of experiments to be performed and 
the type of samples to be tested, as well as what is to be defined before 
the process is implemented in casework. Although the results of the 
developmental validation were not published in a peer-reviewed 
journal, according to the FLS the results were provided to the scientific 
community in presentations at scientific meetings and conferences. 
However, the validation studies were conducted by individuals who 
themselves had been involved in designing and implementing the 
automated process. In our view, while the FLS may have met 
SWGDAM validation guidelines, a new process of this significance 
should have been subjected to an external peer review to provide 
independent assurance to management prior to implementation. 


7.72 Of equal concern was the lack of documented sign-off by the 
National Program Manager, the Chief Scientific Officer, and the FLS 
Assistant Commissioner before implementation; this was particularly 
necessary given the importance of the switch to automation. Also 
lacking is documentation showing that the FLS put in place a strategy 
for dealing with limitations or making key decisions before it 
implemented the new process. 


7.13 Recommendation. The RCMP should develop standard 
procedures for project planning and implementation, including 
documentation of decisions and sign-off by senior management. 


The RCMP’s response. In 2004, the FLS created the Project 
Management Office to ensure that PMBOK® (Project Management 
Institute) standard project management practices are applied to FLS 
projects. 

The decision-making process for planning, implementation, and sign- 


off of projects will be subject to a more rigorous and detailed process. 
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Client consultation 


Clients lack opportunity to influence how the FLS operates 


7.74 Not only the RCMP but other members of the law enforcement 
community rely on the FLS. Provincial and municipal police and other 
agencies currently submit about half the service requests, and 
prosecutors rely on the results in court. Furthermore, some FLS 
funding comes from agreements with provinces and territories. It is 
therefore important that clients be consulted about their requirements 
and have an opportunity to influence how the FLS operates to ensure 
that their needs are met. 


7.15 In previous audits we recommended that the RCMP establish a 
national forensic advisory committee. In 2000, the RCMP established 
the Forensic Laboratory Services Advisory Group (FLSAG), which 
provides advice and recommendations to the FLS Assistant 
Commissioner. Currently the FLSAG includes individuals representing 
client expectations (the RCMP and municipal police services), 
partnerships (provincial governments), criminal justice, work 
processes, science, and human resources. Given the terms of reference 
and membership, the Advisory Group appears to be designed as a 
source of expertise to which the FLS can turn for assistance in 
operations, rather than a mechanism for informing and consulting with 
clients. It may be advisable to strengthen the role of the FLSAG, 
making it more of a consultative body. 


7.76 A national forensic advisory committee has been created but the 
intent of our previous recommendations has not been addressed. The 
RCMP still needs to provide an opportunity for client input. 


1.11 The ELS keeps clients informed to some extent through client 
visits, newsletters, written correspondence on specific topics, and the 
RCMP website. However, only about one third of the clients we 
interviewed said that the FLS was good or very good at informing 
clients about operations, decisions, and performance (Exhibit 7.10). 
Furthermore, 42 percent said that they or their agency had received a 
client visit from the FLS. 


7.18 More important, clients reported little consultation on key 
topics. Less than a quarter of the clients we interviewed said that the 
FLS was good or very good at consulting about client needs and 
changes being planned. FLS officials told us that Regional 
Consultation Committees were created in fall 2006. 
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Exhibit 7.10 A small proportion of clients said the FLS was good or very good at informing or 
consulting with them 


Informing about FLS operations 


Informing about FLS performance 


Informing about FLS decisions 


Consulting about agencies’ needs 


Consulting about changes to FLS 


0 5 10 15 20 25 30 35 40 45 
Percentage of clients interviewed 


7.79 Recommendation. The RCMP should establish a mechanism 
for consulting with clients so that they have an opportunity to 
influence lab services, priorities, and service standards. 


The RCMP’s response. The RCMP agrees that client consultation is 
an important component in identifying FLS priorities and ensuring 
client service is aligned with needs. Client consultation groups have 
been established, and a more comprehensive consultative plan will be 
developed to enable clients to influence decision-making processes. 


The FLS will undertake a more active communications approach with 
clients to enhance mutual understanding. 


The FLS does not usually negotiate the due date for service requests 


7.80 In their testimony to the House of Commons Standing 
Committee on Justice and Human Rights, Public Safety and Emergency 
Preparedness in 2005, RCMP officials stated that the FLS negotiates 
due dates with its clients and that each request is negotiated to the 
client’s satisfaction. When asked who decides whether a service request 
is urgent, officials said that the final decision rests with the police 
service and the investigator submitting the case to the lab facility. 


7.81 We examined the extent to which the FLS negotiates the priority 
of a service request and response times with clients. The due date was 
automatically set to the default Expected Diary Date (EDD) for 
92 percent of routine service requests in the 2005-06 fiscal year. 
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Performance reporting 


Routine service requests for which the due date was not simply the 
EDD included requests in cases having a set court date (6 percent) and 
flagged requests (2 percent). The FLS told us that due dates can be 
negotiated but negotiations are infrequent and limited, since deviation 
from the EDD can create problems in the orderly processing of service 
requests nationwide. 


7.82 Our client interviews confirmed the lack of negotiation. In the 
case-specific interviews, 82 percent of respondents said that there was 
no negotiation of the original due date. When asked about specific 
cases, 20 percent said that the due date did not meet their needs. Of 


that 20 percent, two thirds said that the due date set by the FLS had 
affected the arrest or court proceedings. 


7.83 As well as limited opportunities to negotiate turnaround time, 
clients have little say in the number of exhibits they can submit to the 
FLS for analysis. Recent FLS policy allows investigators to submit up to 
eight exhibits with each Biology service request. In rare instances that 
policy may be waived. The FLS says that the limit on the number of 
exhibits enables laboratory staff to better manage service requests and 
improve response times. 


Performance reporting has not improved 


7.84 In our 2000 audit we found that FLS performance measurement 
systems were weak. Stakeholders had little information on the level of 
service at laboratories; laboratory management did not have the 
required information on efficiency. The lack of information was 
reflected in the RCMP’s performance reports, which contained 
discrepancies in data. We recommended that the RCMP ensure full 
implementation of its information management system and improve 
performance reporting. 


7.85 In January 2003 the FLS implemented a new Laboratory 
Information Management System (LIMS). Our review found that 
LIMS captures information needed to monitor operations and respond 
to client needs, and it allows the FLS to operate as a single lab system. 
It is capable of integrating evidence tracking, analytical results, and lab 
management information to provide a clear overview of each case. 
Furthermore, it facilitates the use of a national case numbering system 
and monitoring of the completion of service requests; this allows staff 
to enter case information and track exhibits during the lifetime of a 
case. The FLS regularly adds new features to update and improve the 
system. We found evidence of one significant issue: staff can and do 
overwrite the due date for a request. This is problematic because one of 
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the key FLS performance indicators is the percentage of due dates met. 
If the due date can be overwritten, performance may appear better 
than it actually is. 


7.86 Although LIMS has increased the reliability of information and 
improved the ability to report on performance, the RCMP is not 
reporting FLS performance externally to clients or Parliament. In 2001 
the RCMP accepted the recommendation of the House of Commons 
Public Accounts Committee that it “begin to report financial and 
performance information with regard to its forensic laboratories in its 
annual Departmental Performance Reports, beginning with the Report 
for the period ending 31 March 2002.” The RCMP reported FLS 
performance in its Departmental Performance Report for the 2001-02 
fiscal year, but the report for the 2002-03 fiscal year contained limited 
information on the FLS, and the 2003—04 report contained none. FLS 
performance is currently not reported through any other channel. 
Therefore, little performance information about the FLS is available 
externally. 


1.87 Recommendation. The RCMP should ensure that 
parliamentarians receive the information needed to hold the 
government to account for the performance of all activities related to 
the Forensic Laboratory Services, including information on turnaround 
times and the extent to which performance targets are met. 


The RCMP’s response. The RCMP agrees with government’s 
commitment to transparency and accountability. It will explore 
mechanisms to report to Parliament more fully. 


Conclusion 


7.88 In examining statements made to Parliament by the government 
and the RCMP since our 2000 audit, we found large discrepancies with 
our findings from this audit (Exhibit 7.11). Our first audit objective 
was to determine whether the RCMP delivered timely forensic services 
as claimed in statements by RCMP officials to parliamentary standing 
committees. The RCMP has met some criteria but overall it is not 
delivering timely forensic services according to the targets it has set. 
Although a system for prioritizing requests is in place, the definition of 
“urgent” has been revised so that only | percent of service requests fall 
into that category, compared with 8 percent in the 2003-04 fiscal year. 
Turnaround targets exist for urgent and routine requests, but in most 
instances the Forensic Laboratory Service (FLS) cannot meet the 


targets. Furthermore, we found problems in workload management: 
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the backlog of Biology requests in particular is increasing, as is the 
turnaround time. The average turnaround time for a Biology service 
request is 114 days, up from 91 days in the 2003-04 fiscal year. 


7.89 We examined whether the RCMP had adequate quality control 
of lab results. A national quality management system is in place, 
including accreditation and proficiency testing processes. In practice, 
however, there are significant weaknesses in how the FLS defines, 
monitors, and resolves quality issues. Problems that meet the definition 
of a quality issue were not recorded as such. In addition, the quality 
management system failed to identify problems with the automated 
DNA process. Given these major shortcomings, overall the RCMP 
cannot be said to perform satisfactorily in this area. 


7.90 We looked at whether clients are provided with adequate 
opportunities for influencing how the FLS operates to meet their 
needs. The RCMP met our criteria with regard to establishing a 
forensic advisory group, and it attempts to inform clients through 
client visits and newsletters. According to clients, however, there is not 
much consultation on important topics such as services, priorities, 
timeliness of service, or performance. Clients told us they have little 
opportunity to negotiate the turnaround times for service requests, and 
they have little say in how many exhibits they can submit per service 
request—the maximum number allowed is eight. 


7.91 Finally, we examined whether the RCMP has kept its 
commitment to report on FLS performance to Parliament. The RCMP 
partially met our criteria by implementing a laboratory management 
system. But although it is able to examine performance, the RCMP is 
not reporting on FLS performance to clients or to Parliament. 


7.92 Many of our recommendations focus on addressing weaknesses 
in the FLS internal management systems. Our findings suggest the 
need for standards and indicators so that management can adequately 
measure and report on performance. Similarly, there is a need to 
identify and resolve quality issues systematically. Finally, the FLS needs 
the capability to examine its capacity and efficiency. 
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Exhibit 7.11 Summary of findings on statements to Parliament 


i Sose Government and RCMP statements 


es 


Solicitor General of Canada, Government Response to the 17th Report of the Standing Committee on Public Accounts: RCMP— 
_ Services for Canada’s Law Enforcement Community, June 2001 


Our findings 


"It is anticipated that new performance and service standards 
will have been validated and client consultation completed by 
fiscal year 2002. It is the intention of the RCMP to publish 
these standards and to report annually on its performance 
against these standards.” 


* No evidence that performance and service standards were 
validated. 


* New standards have not been published. 


* FLS performance discussed in departmental performance 
reports for fiscal years 2001-02 and 2002-03, but not 
thereafter. 


Standing Committee on Justice, Human Rights, Public Safety and Emergency Preparedness, November 2004 


“There is no backlog in the system. What we have is cases in 
process. There isn’t one major case that is not done within 
15 days. There is no country in the world that meets that 
standard.” 


¢ We examined “cases in process” that had not been 
completed within 30 days (defined as backlog). Overall 
backlog was 2,017 service requests in March 2006. 


* Most urgent service requests are completed within 15 days. 
However, the definition of an urgent request was narrowed in 
2003, with the result that 1 percent or fewer service 
requests currently fall into this category. The remaining 
routine service requests include murder and other violent 


offences (38 percent of all requests concern violent offences). 


¢ The UK's Forensic Science Service meets the target of 7 days 
for completing DNA analysis of crime scene stains. 


“Every single major case that requires an emergency DNA 
analysis is done in this country and it’s done within 15 days. 
Every other case is negotiated and discussed with the police 
force or the agency that wants that done, and every single one is 
satisfied.” 


| 


* 82 percent of clients interviewed said there was no 
negotiation of the original due date. 


* 92 percent of routine service requests are given the default 
due date. 


¢« When asked about specific cases, only about one half of 
clients said timeliness was “good” or “very good.” 


“We today on major crimes guarantee and have produced a 
15-day turnaround, which is as good as if not better than 
anywhere else in the world.” 


“(Regarding restructuring,] there is absolutely no loss or 
diminution of service. The timelines we are committed to with 
DNA and other tests stay the same.” 


“Clearly, murder cases, cases with violence, violent assault 
cases, those kinds of things would definitely go right into the 
priority queue and be handled right away.” 
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¢ Analyses for most major crimes are not completed within 
15 days. The FLS categorizes 99 percent of service requests 
as routine, including murder and other violent offences. 


¢ Although the RCMP established turnaround targets of 
15 days for urgent service requests and 30 days for routine 
ones, it has created much longer Expected Diary Dates 
(EDDs), which are provided to clients as an estimate of when 
the examination will be completed. Furthermore, the EDD for 
Biology is currently 180 calendar days, up from 70 days in 
February 2003. 


¢ The timelines for completing Biology service requests have 


increased from an average of 91 days in the 2003-04 fiscal 
year to 114 days in 2005-06. 


¢ 38 percent of service requests relate to violent offences. The 


FLS categorizes most of these as routine and does not give 
them any special priority. 
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Exhibit 7.11 Summary of findings on statements to Parliament (Continued) 


Government and RCMP statements 


“By 2005 we will have substantially improved our ability to 
provide world-class service.” 


In the 2005-06 fiscal year, turnaround times for Biology 


~ Our findings 


requests were longer than in earlier years and backlogs had 
increased. 


“We have a negotiated agreement with the police forces on how 
much time they need: when does their case have to go to court? 
We negotiate to their satisfaction, each one.” 


82 percent of clients interviewed said there was no 
negotiation of the original due date. 


20 percent of clients said the original due date did not meet 
their needs. 
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About the Audit 


Objectives 


We wished to determine whether 


+ the Forensic Laboratory Services (FLS) of the Royal Canadian Mounted Police (RCMP) is delivering 
timely forensic services, 


+ there is adequate quality control of lab results, 
- clients have adequate opportunity to influence how the FLS operates, and 


* the RCMP has kept its commitment to report on FLS performance to Parliament. 


Scope and approach 


The audit was limited to the FLS. We did not examine other RCMP functions or services, nor the National 
DNA Data Bank. In addition, we did not audit the quality of the forensic science; we offer no opinion, 
positive or negative, on the validity of scientific analysis used. 


For all lines of enquiry we reviewed RCMP documents and records, and met with management and staff. 
We also interviewed 42 staff members, 6 members of the Forensic Laboratory Services Advisory Group, 
and representatives from the Standards Council of Canada. We visited eight other labs to gather 
information: 


ine 58 > bahoratory 
“Ontario rs o — ie ponents aoe 
Quebec Laboratoire de sciences judiciaires et de médecine légale 
North Carolina, United States State Bureau of Investigation Crime Laboratory 
; Georgia, United States | Georgia Bureau of Investigation, Division of Foeiei Stee : 
“Pods, United States | Department of Law Enjorcomant. Cine Laboratory —_ 
United Kingdom - 7 — : Forensic Science Service la ie ra 
Sweden: > = ea | National Laboratory of Forensic Selenca 2 =e 
Nethaitands : : | Netherlands Porensi¢ institute 


During our visits we met with senior management and gathered information on lab features such as 
structure, business delivery model, budget, staff, workload, and turnaround times. We relied on 
information either published in the organizations’ reports or confirmed by senior officials. The information 


is unaudited but has been verified by the labs we visited. 
The examination also included a client survey using two types of interviews: 
* case-specific interviews—questions about specific cases on which the FLS conducted analyses, and 


* general interviews—questions about consultation and service in general. 
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For the case-specific interviews we extracted a random sample of 125 cases from the FLS Laboratory 
Information Management System (LIMS) database for which the FLS had completed at least one service 
request associated with a primary offence during the 2005-06 fiscal year (representing 2,117 cases out of 
8,993). Ten of the 125 cases selected were found to be invalid; that is, we could not trace the case file 
number, or the investigator responsible for the case had retired or was unavailable during the survey, and 
no one else could answer our questions. For the 115 remaining cases identified, we interviewed 

113 individuals: 65 RCMP and 48 municipal police investigators. The response rate was 98.3 percent. 
Overall estimates are accurate to plus or minus 6.3 percent 18 times out of 20. 


We used two methods to develop a sample for the general interviews. 


- Investigators: We used the 115 cases randomly selected for case-specific interviews. We asked the chief 
of police (municipal) or the unit supervisor (RCMP) associated with each case to put us in contact 
with the person most familiar with the FLS. We interviewed 86 of the 92 individuals contacted, 
including 46 RCMP and 40 municipal investigators. 


e 


Prosecutors: From the LIMS database we drew a random sample of 60 prosecutors with three or more 
service requests over the last three years. Of these, 4 were considered to be invalid. We interviewed 49 
of the remaining 56 prosecutors. 


In total, we conducted 135 general interviews (with 86 investigators and 49 prosecutors), for a response 
rate of 91.2 percent. Overall estimates are accurate to plus or minus 5.8 percent 18 times out of 20. 


To examine timeliness we downloaded data from LIMS and examined three fiscal years: 2003-04 
through 2005-06. We analyzed data using the Statistical Package for the Social Sciences to confirm the 
number of cases and service requests, turnaround times, processing and waiting times, and backlog. We 
used the service request as the main unit of analysis because each case may include more than one service 
request with different due dates. 


Criteria 

We expected to find the following: 
- Information provided to Parliament was complete and accurate. 
- A prioritizing system was in place. 
- High-priority cases were handled first. 
- Turnaround targets had been set for initial DNA screening tests. 
- Turnaround targets had been set for DNA analysis. 


- Turnaround targets had been validated with clients, and were based on operational requirements and 
generally accepted international standards. 


- Performance against these targets was monitored and corrective action taken. 
- Turnaround targets established by the FLS were met. 


* Turnaround times had improved relative to workload and resources. 
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- Turnaround times achieved by the FLS were reasonable in relation to benchmarked standards. 
» Discretionary activity had been reviewed and budgets re-allocated. 
- Accreditation of all sites was complete and up to date. 


- A quality assurance program and appropriate procedures were in place to ensure quality for new 
initiatives and processes. 


- A proficiency testing program was in place and adequately monitored. 


* A national forensic advisory committee had been established, and was representative of the population 
of clients and stakeholders being served. 


* Clients were being consulted and informed about FLS procedures and processes. 
- Response times were being negotiated with clients and agreed delivery dates were being met. 
+ An information management system had been fully implemented. 


- Performance reporting had improved. 


Audit work completed 


Audit work for this chapter was substantially completed on 20 November 2006, and was updated to reflect 
further information received up to 16 February 2007. 


Audit team 


Assistant Auditor General: Hugh McRoberts 
Senior Principal: Peter Kasurak 
Director: Shelley Trevethan 


Claudette Blair 
Alexis Dusonchet 
Tara Beauchamp 
Tamara Jensen 


For information, please contact Communications at 613-995-3708 or 1-888-761-5953 (toll-free). 
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Appendix A Recommendations arising from past audits of forensic services 


1990 Report of the Auditor General, Chapter 27, Royal Canadian Mounted Police—Support Services to Canadian Law 
Enforcement Agencies 


* FLS should take steps to prevent obvious non-criminal cases from being submitted for examination. 


° FLS should ensure that future laboratory construction decisions are based on the identification and 
cost/benefit analysis of all viable options. 


* FLS should improve its performance measurement system by using an appropriate unit of 
measurement, recording data daily, and developing accurate standards. 


April 2000 Report of the Auditor General, Chapter 7, Services for Canada’s Law Enforcement Community 


¢ In consultation with users, the RCMP should establish turnaround targets for initial screening tests for 
DNA warrants and turnaround targets for DNA analysis. Managers should monitor performance 
against these targets and take corrective action to improve service. 


© The RCMP should ensure that priorities are set for all cases and high-priority cases are handled first. 
All discretionary activity should be reviewed and budgets reallocated. 


¢ The RCMP should establish a national forensic advisory committee. 


¢ The RCMP should ensure quality in its laboratories by timely accreditation, documentation of 
methodology, and full implementation of the proficiency testing program. 


¢ The RCMP should ensure that its information management system is fully implemented and its 
performance reporting is improved. 


October 2000 Report of the House of Commons Standing Committee on Public Accounts after considering the April 2000 
Report of the Auditor General 


¢ That the Royal Canadian Mounted Police develop a strategy to rationalize its forensic laboratory 
facilities and include that strategy, along with implementation dates, in its Report on Plans and 
Priorities (RPP) for fiscal year 2002-03. 


¢ That the Royal Canadian Mounted Police develop and implement a series of performance indicators 


and standards for each of the services provided by its forensic laboratories by the end of fiscal year 
2000-01. 


* That the Royal Canadian Mounted Police use the results of performance measurement in conjunction 
with efforts to rationalize its forensic laboratories and consolidate services offered through those 
facilities. 


¢ That the Royal Canadian Mounted Police use data generated by performance measurement to redirect 
resources to the areas, such as DNA testing, where they are most needed. 


¢ That the Royal Canadian Mounted Police begin to report financial and performance information with 
regard to its forensic laboratories in its annual Departmental Performance Reports, beginning with the 


Report for the period ending 31 March 2002. 
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Appendix B List of recommendations 


The following is a list of recommendations found in Chapter 7. The number in front of the 
recommendation indicates the paragraph where it appears in the chapter. The numbers in parentheses 
indicate the paragraphs where the topic is discussed. 


Recommendation 


Timeliness of service 


7.28 The RCMP should ensure that 
the Forensic Laboratory Services’ 
prioritization system and turnaround 
targets meet the operational needs of 
clients. Turnaround targets should be 
implemented before service requests 
increase, as they are expected to do 
once amendments to the Criminal Code 
and the DNA Identification Act come 
into force. The targets should be used 
to measure and report FLS 
performance. (7.20-7.27) 


7.34. The RCMP should develop 
measures of service efficiency and 
effectiveness in consultation with 
clients. In addition to existing tools for 
measuring client satisfaction, it should 
use client surveys conducted by an 
independent, external organization. 


(7.29-7.33) 
7.43. The RCMP should develop 


mechanisms for identifying bottlenecks 
in the process and should determine the 
systems, procedures, and resources 
required to eliminate the backlog. 


(7.40-7.42) 
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The RCMP agrees that response times must be reduced. As the 
report indicates, significant reductions have been achieved in all 
disciplines except Biology (DNA). 


The RCMP agrees that client consultation is important in 
ensuring that response times meet client needs. The RCMP will 
continue to engage its clients to establish reasonable turnaround 
targets. These targets will form the basis of performance 
reporting. 


The FLS acknowledges the significant additional demands on 
Biology services that the proposed changes to the Criminal Code 
and the DNA Identification Act will create. The FLS has 


identified its requirements to meet the proposed changes. 


The RCMP agrees that measurement of efficiency and 
effectiveness, both internally and from client feedback, is 
important. The FLS currently uses a Quality of Service 
Questionnaire, which it encourages clients to complete. The 
RCMP agrees that use of an independent third party to develop 
and receive feedback may improve response rates and reduce 
the likelihood of bias. It will further explore this option. 


The RCMP agrees that identifying bottlenecks in processes is 
important, and will undertake workflow analyses to determine 
further efficiencies. 


Chapter 7 37 


MANAGEMENT OF FORENSIC LABORATORY SERVICES—ROYAL CANADIAN MOUNTED POLICE 


7.50 The RCMP should conduct a 
review of the Forensic Laboratory 
Services to examine internal 
efficiencies, perform a cost/benefit 
analysis of various services, and 
examine the need for additional 


resources. (7.44—7.49) 


7.51 The RCMP should develop a 


capability for management of the 


Forensic Laboratory Services to analyze 


capacity and efficiency (including 
comparing performance with that of 
other forensic labs) in order to handle 


future demands. (7.44—7.49) 


As per recommendation 7.43, the RCMP agrees that there 
should be cost/benefits analyses for all services and will 
undertake a workflow analysis to determine if there are further 
operational efficiencies to be gained. 


The RCMP agrees that benchmarking is an important tool to 
measure performance and analyze capacity to meet future 
demands. The FLS will strive to balance capacity with 
performance by developing individual and unit performance 
metrics, and ensure that they are measurable and reportable. 


Quality of lab results 


7.68 The RCMP should take measures 


to ensure identification of all quality 
issues. The RCMP should ensure that 
quality issues, once identified, are 
systematically tracked and resolved, 
and made available in a consolidated 
form, and that actions are 


communicated to senior management. 


(7.52-7.67) 


7.13. The RCMP should develop 
standard procedures for project 
planning and implementation, 
including documentation of decisions 
and sign-off by senior management. 


(7.69-7.72) 


The FLS agrees that it will put into place managerial 
mechanisms to ensure that quality issues are defined clearly, are 
recorded and tracked in a systematic manner in its Quality 
Management System, are available in a consolidated format, 
and that all actions are communicated to senior management. 


In 2004, the FLS created the Project Management Office to 
ensure that PMBOK® (Project Management Institute) standard 
project management practices are applied to FLS projects. 


The decision-making process for planning, implementation, and 
sign-off of projects will be subject to a more rigorous and detailed 
process. 
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Client consultation 

7.79 The RCMP should establish a The RCMP agrees that client consultation is an important 
mechanism for consulting with clients component in identifying FLS priorities and ensuring client 

so that they have an opportunity to service is aligned with needs. Client consultation groups have 
influence lab services, priorities, and been established, and a more comprehensive consultative plan 
service standards. (7.74-7.78) will be developed to enable clients to influence decision-making 


processes. 


The FLS will undertake a more active communications approach 
with clients to enhance mutual understanding. 


Oe es de ee 
Performance reporting 


7.87 The RCMP should ensure that The RCMP agrees with government’s commitment to 
parliamentarians receive the transparency and accountability. It will explore mechanisms to 
information needed to hold the report to Parliament more fully. 

government to account for the 

performance of all activities related to 

the Forensic Laboratory Services, 

including information on turnaround 

times and the extent to which 

performance targets are met. 


(7.84—7.86) 
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